Please support out-of-tree builds in manifest mode

[omartijn]

I'm using vcpkg in manifest mode, where we have the vcpkg repository checked out as a submodule. During the configure step (where vcpkg will download and build dependencies), I'm running into a number of issues:

• doing parallel configures often fails
• an out-of-tree build using a read-only checkout fails

Both issues have to do with the fact that vcpkg clobbers its own checkout. The first goes wrong like this:

• configure for debug starts downloading vcpkg executable (with a .part extension)
• configure for release starts downloading vcpkg executable (with a .part extension)
• both downloads finish roughly at the same time
• both configures want to do a chmod and rename (remove the .part extension)
• one of the two fails chmod because the other has already renamed

It would all be so much better if vcpkg would just write to the CMAKE_BINARY_DIR when running in manifest mode. Each configure gets its own CMAKE_BINARY_DIR and these kind of race conditions just won't happen.

Another issue this causes is when you do a build inside a docker container where you bind-mount the repository into the container. That way you end up with a lot of files owned by root inside your submodule. If I do a read-only bind-mount it will not configure, of course.

I've tried setting the following:

• VCPKG_DOWNLOADS (environment variable)
• --buildtrees-root (via VCPKG_INSTALL_OPTIONS)
• --packages-root (via VCPKG_INSTALL_OPTIONS)

This has eliminated most of the writes to the vcpkg submodule, but I cannot find a way to stop vcpkg from writing the vcpkg executable itself into the vcpkg submodule. I've tried different things - like setting VCPKG_ROOT and Z_VCPKG_ROOT, but I cannot get it to work.

Additionally, I'd argue that it makes sense that there would be a single, simple option, to tell vcpkg to just write all its stuff in the CMAKE_BINARY_DIR. I'd even argue that that should be the default option, but I'd settle for just getting this to work at all.

[WangWeiLin-MV]

It would all be so much better if vcpkg would just write to the CMAKE_BINARY_DIR when running in manifest mode. Each configure gets its own CMAKE_BINARY_DIR and these kind of race conditions just won't happen.

Please try adding option https://learn.microsoft.com/en-us/vcpkg/maintainers/functions/vcpkg_cmake_configure#disable_parallel_configure. If it is not a private port, please tell us to fix it.

---

I've tried setting the following

You could try the command option https://learn.microsoft.com/en-us/vcpkg/commands/common-options

--vcpkg-root # the vcpkg instance, aka the git repository
--downloads-root
--x-buildtrees-root
--x-packages-root
--x-install-root

---

writing the vcpkg executable itself

The executable could be download in https://github.com/microsoft/vcpkg-tool/releases

---

Additionally, I'd argue that it makes sense that there would be a single, simple option, to tell vcpkg to just write all its stuff in the CMAKE_BINARY_DIR. I'd even argue that that should be the default option, but I'd settle for just getting this to work at all.

At least at present, vcpkg binary depends {VCPKG_ROOT}/scripts, making it difficult to separate the git repository as an completely independent component.

As a workaround, can pass the required commit into docker. And if there are custom port, please see https://learn.microsoft.com/en-us/vcpkg/concepts/overlay-ports

[omartijn]

It would all be so much better if vcpkg would just write to the CMAKE_BINARY_DIR when running in manifest mode. Each configure gets its own CMAKE_BINARY_DIR and these kind of race conditions just won't happen.

Please try adding option https://learn.microsoft.com/en-us/vcpkg/maintainers/functions/vcpkg_cmake_configure#disable_parallel_configure. If it is not a private port, please tell us to fix it.

Why? It breaks during downloading of tools, like the vcpkg executable. If multiple configure jobs are run in parallel, this will still happen multiple times. It's completely unrelated

I've tried setting the following

You could try the command option https://learn.microsoft.com/en-us/vcpkg/commands/common-options

--vcpkg-root # the vcpkg instance, aka the git repository
--downloads-root
--x-buildtrees-root
--x-packages-root
--x-install-root

writing the vcpkg executable itself

Yes, I know that. But that's not what we do. We depend on the vcpkg manifest scripts to download the tool.

The executable could be download in https://github.com/microsoft/vcpkg-tool/releases

Additionally, I'd argue that it makes sense that there would be a single, simple option, to tell vcpkg to just write all its stuff in the CMAKE_BINARY_DIR. I'd even argue that that should be the default option, but I'd settle for just getting this to work at all.

At least at present, vcpkg binary depends {VCPKG_ROOT}/scripts, making it difficult to separate the git repository as an completely independent component.

As a workaround, can pass the required commit into docker. And if there are custom port, please see https://learn.microsoft.com/en-us/vcpkg/concepts/overlay-ports

What do you mean with "passing the required commit into docker"?

[WangWeiLin-MV]

What do you mean with "passing the required commit into docker"?

I mean just "bind-mount" the required content rather than the whole git repository. Then you can git checkout the same commit in containers.

docker run -e MY_VARIABLE=myvalue myimage

[omartijn]

That's just a very inconvenient workaround. Cloning the repository takes about 20 minutes, then lfs adds another 10, so a simple build job in a docker will now take several tens of gigabytes more and about 30 minutes longer.

[WangWeiLin-MV]

writing the vcpkg executable itself

The executable could be download in https://github.com/microsoft/vcpkg-tool/releases

Hope this could solve the issue.

[omartijn]

writing the vcpkg executable itself

The executable could be download in https://github.com/microsoft/vcpkg-tool/releases

Hope this could solve the issue.

It does not. At best, it'd be a workaround. It's not even that, however, since all the scripts that vcpkg comes with will assume the executable to be inside the vcpkg submodule, which it shouldn't be. So even replicating all the scripts to get the correct release matching the commit we have of vcpkg (it's set somewhere in one of the scripts), figuring out the architecture, downloading it, would not be enough to get this to work.

[Neumann-A]

Consider using the vcpkg-init.ps1 script and just adjust .vcpkg-root to whatever. Then setup registry lookup so that you don't need a full vcpkg checkout of the registry. However, the CMake script could also take a lock on Z_VCPKG_ROOT_DIR before bootstrapping in vcpkg.cmake.

[omartijn]

I'm note quite how that'd work. There is no vcpkg-init.ps1 script in the repo. I see it's there in the vcpkg-tool repo, but I have no idea how I would execute a ps1 script. It still seems unnecessarily complex for doing something that should IMNSHO work out of the box.