Package installation layout for tools

[dg0yt]

I fail to find documentation (or a specification) for the package installation layout.

In particular, I would like to know where to properly place tools like curl-config or geos-config, i.e. executable scripts which provide information about how to use the corresponding libraries.
Software which uses theses script (e.g. gdal) expects them to be found by name via the PATH environment variable or to be provided explicit by parameters (e.g. gdal's --with-geos)

This is what I found in vcpkg:

• The release variant of these scripts (curl-config, geos-config) is currently installed to
  share/${PORT}
  The debug variant (which could provide the d-suffixed library names) is dropped,
  Fixed.
• The following locations are automatically added to the PATH environment variable by vcpkg.cmake, even for cross-builds (Android, iOS)(since [vcpkg.cmake] Setup CMAKE_PROGRAM_PATH for the host if possible #23322:) for the host triplet:
  tools
tools/*
  (but no extra subdirectories) and all subdirectories (since [vcpkg.cmake] Setup CMAKE_PROGRAM_PATH for the host if possible #23322)
• For all declared dependencies, the following locations are automatically added to the CMAKE_PROGRAM_PATH cmake variable:
  tools\<DEPENDENCY>
  This variable is used by cmake_find_program which is sometimes called with PATH_SUFFIXES bin (FindMPI.cmake [openmpi] find_package(MPI) doesn't work correctly for debug configuration #18151), sometimes without suffixes (FindGettext.cmake [gettext] msgfmt (and other gettext bin tools) missing on macOS & Windows #13518).
• sqlite3 executable is installed in
  tools
• vcpkg_copy_tools defaults to dealing with
  tools/${PORT}
• vcpkg_configure_make installs tools to
  tools/${PORT}/bin
tools/${PORT}/sbin
tools/${PORT}/debug/bin
tools/${PORT}/debug/sbin
• vcpkg_configure_meson has undocumented options ADDITIONAL_NATIVE_BINARIES, ADDITIONAL_CROSS_BINARIES which are used by some ports like
  tools/${PORT}/<COMPONENT>
• vcpkg_cmake_config_fixup adjusts the path of tools references in cmake config files to
  tools/${PORT}
• dbus installs debug tools to
  debug/tools/${PORT}.
• Some tools which shall not automatically be in the (CMAKE_PROGRAM_)PATH are installed to
  manual-tools/${PORT}.
  The Linuxish standard for this would be
  libexec
  and it is used, too.

My questions:

1. When to use tools and when to use another prefix?
2. Where to place debug variants, if needed?
3. Where to place tools created for the host during a cross-build, vs. tools created for the target?
   ANSWER: Host tools must be built in the host triplet only.
   COMMENT: This can create bootstrapping problems, example: luajit (target -> host -> target), qt5-base (qmake and mkspecs).
4. Shouldn't the modification of the PATH environment be restricted to explicitly declared (host) dependencies instead of globbing tools/*?

[Neumann-A]

To add to this: Qt installs into tools/qt5/bin due to the Qt VS integration being stupid and expecting qmake in <prefix>/bin (instead of parsing qt.conf or accepting a qt.conf.....). I have seen a few more cases where tools have been expected to be in <prefix>/bin which is why vcpkg_configure_make installs there.

[dg0yt]

expecting qmake in /bin

Hm, Qt configure has a got number of options, including a separate --host-prefix which helps to deal with cross builds.

But more generally, it is standard to have binaries in <prefix>/bin. I think it is fair to add:

5. Why doesn't vcpkg simply use the bin prefix?

[dg0yt]

Related: #13649

[dg0yt]

I would really appreciate some guidance or perspective, and it seems that I'm not alone.

[NancyLi1013]

@ras0219-msft
Could you please help take a look? thanks.

[dg0yt]

I continued updating the findings in the initial post. I just added CMAKE_FIND_PROGRAM_PATH behaviour which does not cover debug variants.

[dg0yt]

Friendly reminder...

[dg0yt]

Another related issue: Is there a good reason why the vcpkg cmake toolchain is not adding (/debug)/bin to PATH? #7826, now #18344.

[dg0yt]

Three more weeks without feedback to this issue. Not to mention the previous issues. Hm.

[PhoebeHui]

@dg0yt, sorry for the delay, I will talk this issue again in our tomorrow meeting.

[ras0219-msft]

Sorry for the long delay; you've asked a very detailed question and the truth is that we do not currently have guidance to the level of strictness and precision implied by the question!

In general, vcpkg's policies around tools are informed by the idea that vcpkg is first and foremost a C/C++ library dependency manager. We are strongly against growing into a general purpose package manager containing things like vim, apache, or toolchains. Unfortunately, there is no 100% clean line of separation between these things -- we can always imagine adding just a little bit of scope creep to get more functionality at any point on the spectrum, eventually resulting in apt et al.

When to use tools and when to use another prefix?

Our general intent right now is that tools/${PORT} is generally the correct place for tools, unless the port has a separate reason it might want to put them into subfolders or siblings (tools/${PORT}/tool1 or tools/${PORT}-tool1).

We do not have a wholistic solution for curl-config, save that we have a strong preference for consumers inside and outside vcpkg to use pkg-config or CMake instead.

For all declared dependencies, the following locations are automatically added

Shouldn't the modification of the PATH environment be restricted to explicitly declared (host) dependencies instead of globbing tools/*?

Yes, with host dependencies this needs attention (though we must maintain backwards compatibility).

Where to place debug variants, if needed?

Generally, a need for debug variants violates the design model. If pressed, they would go in debug/tools/, but I would strongly avoid that if at all possible.

Where to place tools created for the host during a cross-build, vs. tools created for the target?

vcpkg's build model is that only one toolchain is available at a time, so it's "not possible" for a single build to produce both. If, for some reason, a build did produce both, the package would be responsible for only deploying the requested subset (probably both if native compiling, but only the target tools if cross-compiling).

[dg0yt]

I feel that the problem with the current situation is not fully understood.
There should be:

• one pattern for installing release tools.
• one pattern for installing debug tools.
• a mechanism for adding the path of installed tools (dependencies) to the PATH when building a port.

With regard to the debug variant, I don't ask for more than having a place to install the debug variant of curl-config etc. I don't need a debug variant of msgmerge.
I need neither release nor debug variants of vim (from vcpkg). The original post intentionally only lists tools that are needed to build software

[wrobelda]

To add to this: Qt installs into tools/qt5/bin due to the Qt VS integration being stupid and expecting qmake in <prefix>/bin (instead of parsing qt.conf or accepting a qt.conf.....). I have seen a few more cases where tools have been expected to be in <prefix>/bin which is why vcpkg_configure_make installs there.

I just got affected by it: #19203

The following locations are automatically added to the PATH environment variable by vcpkg.cmake, even for cross-builds (Android, iOS):
tools
tools/*
(but no extra subdirectories)

But more generally, it is standard to have binaries in /bin. I think it is fair to add:
Why doesn't vcpkg simply use the bin prefix?

Couldn't just tools/*/bin be also added to PATH? Since we're already globbing tools/* opportunistically, also checking the /bin subfolders makes sense, doesn't it?

[dg0yt]

One more point on tools/debug/bin: For ports based autotools/vcpkg_configure_make, the debug variant of tools is always built and installed. Is this needed, or shall portfile maintainers try to avoid this?
For port gdal, with static linking as only option on linux, on-disk size is 6.5 GB, with 4.9 GB alone for the unstripped binaries in tools/gdal/debug/bin.

[JackBoosY]

Maybe we should write a document about that.

[autoantwort]

@ras0219-msft Can we at least explicitly prohibit debug executables in tools/?

We could add a post build lint check for that

[dg0yt]

@ras0219-msft Can we at least explicitly prohibit debug executables in tools/?

We could add a post build lint check for that

I am not sure. We do need some config tools from the debug variant.

[dg0yt]

So Microsoft is now promoting debug/tools as another location for tools: #37736. 🔥

[Neumann-A]

So now we could install a debug python interpreter to the same location

[Osyotr]

So now we could install a debug python interpreter to the same location

Does cmake even support finding debug interpreter?

It really is a bummer to have yet another tools directory :(

[dg0yt]

We could also stop removing debug/include and help fix debug-only triplets.

[dg0yt]

It really is a bummer to have yet another tools directory :(

.. and at the same time to explain that "expanding the definition of where debug tools should go depends on the specific use cases and we don't know of any generic use cases at this time."

[ras0219-msft]

I'd like to update #17607 (comment) given everything we've learned in the past ~3 years.

In general, vcpkg's policies around tools are informed by the idea that vcpkg is first and foremost a C/C++ library dependency manager. We are strongly against growing into a general purpose package manager containing things like vim, apache, or toolchains. Unfortunately, there is no 100% clean line of separation between these things -- we can always imagine adding just a little bit of scope creep to get more functionality at any point on the spectrum, eventually resulting in apt et al.

I feel that a lot of confusion is introduced into this discussion by lumping every executable file into the same label of "tools" and seeking a single policy decision to cover everything -- intent and restrictions unfortunately matter. I'm aware of several quite different use cases for executables, each of which may have independent deployment locations and policies.

1. End-user applications that are generally run via a "double click" or interactively. For example, vim.
2. System services that are often singletons. For example, httpd/Apache or dbus.
3. Code generators which have a version-coupled runtime. For example, protoc/Protobuf.
4. Configuration utilities that are executed at build time to locate dependencies. For example, pkgconf/pkg-config/curl-config.
5. Executable content that is launched by a library as part of its operations, intended to be deployed with a developer's overall product. For example, dbus-daemon which can be launched by libdbus.
6. Python used to run a build script. Note that this is different than embedding python in an application, which are both different than building a plugin/wheel/native library for python. (This is made extra confusing because CMake has not historically made this distinction in find_package()).

Of the above scenarios, (1) and (2) should not be deployed via vcpkg -- we maintain that we are a very bad replacement for APT/winget/Homebrew and those system package managers should be used instead for those applications. (3), (4), (5), and (6) all have merits within the context of a C++ library dependency manager, because they all are implicated in successfully consuming C++ libraries.

Further, these executables can have unfortunate restrictions on their layout.

1. They may be precompiled with different shared dependency versions than those present in the main installed/<triplet>/lib directory; this is common for programs that expect to app-local their DLL dependencies on Windows.
2. They may have hardcoded reliance on relative paths between multiple executables. For example, qmake insisted on being installed in a folder named bin1.

These restrictions and widely differing use cases make it impossible to create a one-size-fits-all policy around the general concept of "executables".

When to use tools/ and when to use another prefix?

Our general intent right now is that tools/${PORT} is generally the correct place for tools, unless the port has a separate reason it might want to put them into subfolders or siblings (tools/${PORT}/bin or tools/${PORT}-tool1). This could be caused by the restrictions mentioned above of hardcoded required containing folder names or having conflicting DLL versions.

We do not have a wholistic solution for curl-config, save that we have a strong preference for consumers inside and outside vcpkg to use pkg-config or CMake instead.

Why doesn't vcpkg simply use the bin prefix?

We do not generally recommend bin/ (and restrict it by policy) because it

1. Requires that executables/tools must use the shared library versions built in the tree
2. Requires that executables/tools must be happy with living in exactly bin/
3. Requires many consumers to use all or none of the tools from the tree (via adding to PATH). Ideally it would be possible to always pass tools via an absolute path, but in many cases there's an expectation that the tool is able to be found via path lookup.
4. Is extremely easy to accidentally ship and depend upon undesired executables that may be built in a "default" configuration.

The following locations are automatically added to the PATH environment variable by vcpkg.cmake

Shouldn't the modification of the PATH environment be restricted to explicitly declared (host) dependencies instead of globbing tools/*?

Maybe I'm misreading the linked commit, but this is adding them to CMAKE_PROGRAM_PATH, not PATH. This key distinction means that they are available to be found with find_program(), but won't automatically hijack execute_process calls and DLL lookups unless the consuming CMakeLists.txt opts in via adding to path.

For all declared dependencies, the following locations are automatically added

See previous; this makes consumers able to find things via find_program if they're looking for them. Agreed that it would be awesome to restrict CMAKE_PROGRAM_PATH to only contain the dependency subgraph, but that does seem technically challenging to do.

Where to place debug variants, if needed?

Generally, a need for debug variants violates the design model. If pressed, they would go in debug/tools/, but I would strongly avoid that if at all possible. For example, there should not be a need for a debug variant of protoc; the external behavior of the tool should be identical between release and debug.

The few cases I'm aware of that meaningfully need a debug variant would be the libdbus spawning dbus-daemon case (since dbus-daemon is shipped with the final product, it needs to match the final product's build configuration) and possibly the curl-config case.

From #37736 (comment):
...

• debug/tools/... (12 matches in x64-linux CI)
• tools/.../debug/bin (210 matches in x64-linux CI)

but not for

• tools/.../debug (a few matches in vcpkg CI).

I strongly doubt that all of these ~222 matches should be deploying debug variants at all according to our above policy. Regardless of coming to some mega-conclusion about where debug executables should live, ports should not deploy debug versions at all if we can avoid it.

Can we at least explicitly prohibit debug executables in tools/?

We could add a post build lint check for that

I agree, this would be a good post-build lint check (with a suppressing policy flag). As above, the vast majority of ports shouldn't be installing debug executables at all, let alone in the tools/ folder.

Where to place tools created for the host during a cross-build, vs. tools created for the target?

vcpkg's build model is that only one toolchain is available at a time, so it's "not possible" for a single build to produce both. If, for some reason, a build did produce both, the package would be responsible for only deploying the requested subset (probably both if native compiling, but only the target tools if cross-compiling).

Is there a good reason why the vcpkg cmake toolchain is not adding (/debug)/bin to PATH?

I think it's hard to determine the correct set of paths to add, because PATH-based lookup of dependencies means there can be only one winner for every application. This causes problems with DLLs that use the same name in debug and release.

Breaking down the possible scenarios I'm aware of for performing a PATH lookup:

1. I want to run just-built executables as part of my own build without running an applocal process
2. I want to avoid copying dependency DLLs into the tools/ folder of host ports.

For example, if I'm performing a debug, non-cross-compile build, I want to build and run a custom code generator, and I want to run an executable from the tools/ folder, there's no PATH that will satisfy both of these. Either I'll get the release version of libfoo when I try to run my custom code generator or I'll get the debug version of libfoo when I try to run tools/foo/foo.exe.

Fortunately, both scenario (1) and (2) can be worked around: either by running the applocal step as part of post-link before execution or by copying dependencies into the tools/ folder. By doing both workarounds, we remove the need for adding /bin to the PATH entirely.

[Neumann-A]

The few cases I'm aware of that meaningfully need a debug variant

Add QtWebengineProcess(d)?.exe to that.

[ras0219-msft]

QtWebengineProcess(d)?.exe

Is that the same situation as dbus -- the webengine library spawns a separate webengine process for sandboxing?

[Neumann-A]

Is that the same situation as dbus -- the webengine library spawns a separate webengine process for sandboxing?

Probably, somebody requested for it to be deployed so they can run their application in debug configuration. I only run release only builds for stuff like this due to build time.

[dg0yt]

FTR my app comes with Qt Assistant as a portable help viewer.

[tsondergaard]

Is that the same situation as dbus -- the webengine library spawns a separate webengine process for sandboxing?

Probably, somebody requested for it to be deployed so they can run their application in debug configuration. I only run release only builds for stuff like this due to build time.

Yes, QtWebEngineProcess(d) is a good example of another binary executable that is needed at runtime by a library - similar to dbus-daemon and dbus-launch (#37625). QtWebEngineProcess(d) is even a bit worse as it only supports dynamic linking and on linux dynamic linking it will be built with RUNPATH [$ORIGIN:$ORIGIN/../../lib]. That's really annoying as it means applications have to take extra steps to make this work. The consuming application has a few choices:

• Use patchelf to fixup the RUNPATH
• Create a directory structure so the existing RUNPATH works and use either the QTWEBENGINEPROCESS_PATH environment variable or set QLibraryInfo::LibraryExecutablesPath in qt.conf to tell the QtWebEngine library where the executable is

The non-trivial treatment of executables in vcpkg brings a non-trivial amount of pain.