RFC: Package vcpkg-generated binaries as Chocolatey packages?

[traversaro]

Looking into ROS2 effort to package dependency in Windows [1] I noticed that they distribute some of their binary dependencies [2] as nupkg Chocolatey packages, that they manually build using the code in [3].

I wonder if we could modify the scripts in this repo to package the compiled version of vcpkg's libraries as Choco package, so that they could be installed/uninstalled from the chocolatey client. CPack at the moment has no support for generating choco packages [4], and vcpkg has only a limited support to generate "monolithic" Nuget packages [5, 10], so we will need to roll out our own logic for the vcpkg --> choco packaging. However, as far as I understand it should be as simply as:

• Automatically generate the .nuspec file from the vcpkg metadata contained in the CONTROL file (that we already parse in [6] ) and if necessary from a secondary source of metadata.
• Run choco pack to generate the .nupkg file.

This approach would drastically simplify packaging of libraries with a lot of dependecies such as PCL [9], but could present the following challenges/things that need to be properly understood:

• vcpkg currently uses a rolling release model (similar to homebrew) but it could be convenient to just produce binary from a specific snapshot/tag of vcpkg
• How to handle all the different vcpkg triplets, i.e. x86-windows, x64-windows, x86-windows-static, x64-windows-static?
• choco supports the possibility of having a third-party "repository" of packages?
• Some libraries are already packaged in the choco, such as boost [7] or opencv [8]. To avoid API/ABI incompatibilities the best idea would be to prefix in the name of all this vcpkg-generated choco packages the vcpkg- prefix (or even a vcpkg-<something>- to stress the fact that it is not an official vcpkg endeavor) and declare in nuspec metadata (if possible) the incompatibility with this upstream packages.

cc @drdanz @j-rivero

[1] : https://github.com/ros2/ros2/wiki/Windows-Install-Binary
[2] : https://github.com/ros2/choco-packages/releases/latest
[3] : https://github.com/ros2/choco-packages
[4] : chocolatey/choco#655
[5] : microsoft/vcpkg#1035
[6] : https://github.com/traversaro/vcpkg-binary-installer-generator/blob/master/CMakeLists.txt#L97
[7] : https://chocolatey.org/packages/boost-msvc-12
[8] : https://chocolatey.org/packages/OpenCV
[9] : ros2/choco-packages#1
[10] : https://blogs.msdn.microsoft.com/vcblog/2017/05/03/vcpkg-introducing-export-command/

[traversaro]

This is intended to be only used and consumed by downstream projects and not by upstream vcpkg, but perhaps also @ras0219-msft could be interested in this approach.

[ras0219-msft]

We've definitely considered this approach and deemed it a bit too complex and out-of-scope for now -- if you can make it happen cleanly that would be incredible 😄!

Let me lay out some of our reasoning that you might find provocative or useful; apologies in advance for the wall of text!

• Due to ABI instability and the nature of native code, the "effective ABI version"[1] has a transitive nature; it's a hash of { my sources, toolchain, relevant compilation flags, and all of my dependency effective ABI versions }. This properly captures situations such as azure-storage-cpp depends on cpprestsdk which depends on boost, so an update to boost changes the effective ABI of azure-storage-cpp even though neither cpprestsdk's nor azure-storage-cpp's nominal version changed.
  • This implies that, for a sufficiently absurd definition of capable, a fully capable C++ binary package manager would need to build and offer a patently absurd number of package builds to combine all the possible versions. Without the absurd number of packages, you will end up in situations where upgrading package A requires upgrading dozens of other seemingly unrelated packages -- because A@(desired) was not built against B@(current). In many cases, handling the user's "expressed desire" (install A@ver) cannot be handled by a prebuilt package because they want a specific version of B, C, D, ... .
  • Many libraries (especially C libraries) profess to have stable ABIs between some versions (SemVer, etc) which has the potential to dramatically cut down on the combinatoric explosion of builds. Vcpkg explicitly chooses to not trust these because we don't feel the cost/benefit is worth it:
    • Cost: ABI stability must be painstakingly expressed in a form that is intelligible by the package manager. This is nontrivial; you can't just assume SemVer (OpenSSL breaks ABI from 1.0.2 -> 1.1.0 and is one of the most widely used libraries out there). This is a burden on maintainers.
    • Cost: Even once you've gone and cataloged the "official" ABI stance, you still have to deal with human mistakes. ABI is notoriously hard to test and maintain and -- without expressing any distain for the herculean efforts of many library maintainers -- people still get it wrong, which you need to have a process for handling.
    • Cost: When you get it wrong, things can go very wrong. Silent bad codegen, ODR violations, undefined behavior, Access Violations, fire and brimstone, the works.
    • Benefit: You can avoid rebuilding... sometimes. As soon as you have a library that doesn't play the stable-ABI ballgame (boost), you lose all this effort in the downstream cone.
  • I recommend checking out the Nix package manager, which is (afaik) the only package manager to get this full model correct. Vcpkg chose a simpler model (that is still "correct"; just more limited!) because we felt we couldn't build a solid user experience while exposing such a complicated model. As we move into the future, I fully expect us to move more and more in that direction internally -- but while preserving and improving the user-facing experience we currently have.
• Triplets are something of a compromise and should be considered entirely independent worlds of packages; they attempt to represent the "toolchain, relevant compilation flags, and all of my dependency effective ABI versions" from the "effective ABI version" above.
• Chocolatey requires Administrator access to use, which Vcpkg avoids.
• Vcpkg is definitely designed to "snapshot" a particular version -- this is easily accomplished by a git fork or by git submodules. As you noted, we've also added export which we believe sits at a nice balance between the above fraught-with-peril and usefulness.
• Chocolatey can only naturally support a single version with a convenient user experience (in my understanding/opinion), because:
  • When I use #include <boost/asio/asio.hpp> it must resolve to a specific version.
  • To correctly resolve a following #include <cpprestsdk/http_client.hpp>, I must pick up the cpprestsdk package that was built with the boost package found previously.
  • This can be dealt with, but will likely require both
    • An indirection layer in MSBuild, CMake, and every other buildsystem that takes all the desired package versions (from a manifest/lockfile?) into account at once and resolves them to correctly built libraries
    • A user interface layer on top of Chocolatey. I do not believe it is possible to successfully handle install boost@X cpprestsdk@Y in the current implementation -- it will either fail or produce the wrong results.

The conclusion that Vcpkg comes to (regarding all this) is, essentially,

• On any change, rebuild everything from source -- it's far cheaper than getting it wrong
• Always share everything inside a triplet+repo, never share anything between different triplet+repos
• One triplet+repo per process, but many processes per triplet+repo
• Built libraries cannot be meaningfully extracted from the set of all other libraries that are consistent with them

Though it may be obvious, ROS in the "Vcpkg model" would look like:

• Use Vcpkg to build all its dependencies once
• Use extract to make a 7zip or NuGet for each desired triplet (or just a single combined one)
• Distribute the combined archive(s) to users & CI servers
• Rinse and repeat whenever an update is desired (~once a month)

[1] Defining "effective ABI version" such that "all consumers of X will be satisfied with any binary package that has the same effective ABI version"

[traversaro]

Thanks @ras0219-msft for the great insight behind vcpkg's design choices.
The motivation explain well why an "official" repo of vcpkg packages for Chocolatey is not feasible right now.

Just to clarify, my idea on how to use the installer generated by this repo (using the Qt Installer Framework, either online or offline) is quite similar to the "Vcpkg model" that you explained:

• Use Vcpkg to build all the ports you could be interested in, ideally a subset of all vcpkg's repos.
• Build a QtIFW installer for each desired triplet, leaving to the user the freedom to install only the components (ports) he is interested in. Alternatively (and this was the proposal) build a group of Chocolatey packages, and pack them in a single "repo" (I am not familar with Choco's jargon), leaving to the user the freedom to install only the one he is interested in.
• Distribute the QtIFW installer or the repo to users & CI servers
• Rinse and repeat whenever an update is desired (~once a month). Depending on whatever the user are ok with possible ABI breakages in updates, deliver this update as a new installer/a new repo, or provide them as "updates" in the existing installer/repo, to simplify the updating procedure.

The last possibility of having an "update" that is breaking the ABI may seem risky, but typically in systems targeted to developers (such as homebrew), users know that any update in the software will require a complete rebuild of their software. Clearly this system would need to be used only for development, and never for actual deployment in production.

Slightly related: I recently found this discussions on the Homebrew issue tracker on how the deal with versions/ABI compatibility, they are definitely an interesting read: Homebrew/brew#60 , Homebrew/brew#620 , Homebrew/brew#1937 .

[nuclearsandwich]

@traversaro 👋 the work and writing you've done on the Windows packaging front has been really valuable as we explore packaging ROS 2 for Windows.

The Chocolatey packages that you've found in your links [2] and [3] are exploratory and don't necessarily represent the way we plan to package ROS 2 overall. We currently use Chocolatey to install dependencies on our Windows CI build machines so it was natural to try and use it to package additional dependencies as a trial to see how doable generating Chocolatey packages would be.

All I mean by the above is that we aren't committed to using Chocolatey, but we have a community that defies the standard "flavors" of package management strategies. Yes we're installing developer libraries and tools, which Chocolatey doesn't necessarily want to handle but we want to provide those as pre-built binaries for a known set of target systems and we already have tooling for comprehensive source builds which works across platforms and is not . Migrating to or replicating all that work with vcpkg doesn't appear to get us any closer to binary packages for Windows which is my chief concern. vcpkg looks like a great tool, but it's looking likely that it's not the right one for ROS 2's needs.

[traversaro]

Thanks @nuclearsandwich for weighing in the discussion!

All I mean by the above is that we aren't committed to using Chocolatey,

Good to know, we set up this repo exactly for explore how to package a distribute a big group of Windows library binaries with a lot of interdependences, and until we were going toward the idea of packaging them using the Qt Installer Framework and its online capabilities (http://doc.qt.io/qtinstallerframework/ifw-overview.html). Chocolatey was an interesting alternative for which I was curious to hear opinions.

but we have a community that defies the standard "flavors" of package management strategies. Yes we're installing developer libraries and tools, which Chocolatey doesn't necessarily want to handle but we want to provide those as pre-built binaries for a known set of target systems and we already have tooling for comprehensive source builds which works across platforms and is not . Migrating to or replicating all that work with vcpkg doesn't appear to get us any closer to binary packages for Windows which is my chief concern.

I am not sure I am getting your point. I know the extensive work done by the ROS community/OSRF on built tools (in the sense defined by ros2/design#115), but as far as I understand all this effort have typically been used to build a huge group of packages with similar build systems (again, in the sense of ros2/design#115), i.e. the one that then are typically packaged using bloom.

The type of software of which I was thinking are instead the "system dependencies" that are typically installed in Linux using the system package manager or in macOS using Homebrew.
To give you an example, this is the list of the dependencies necessary to build Gazebo on Windows ( complete of secondary dependencies ) :

    boost:x64-windows
  * bzip2:x64-windows
    cppzmq:x64-windows
    curl:x64-windows
    dlfcn-win32:x64-windows
  * double-conversion:x64-windows
  * freeglut:x64-windows
    freeimage:x64-windows
  * freetype:x64-windows
  * gettext:x64-windows
  * glib:x64-windows
  * harfbuzz:x64-windows
  * jxrlib:x64-windows
  * libffi:x64-windows
  * libiconv:x64-windows
  * libjpeg-turbo:x64-windows
  * liblzma:x64-windows
  * libpng:x64-windows
  * libpq:x64-windows
  * libraw:x64-windows
  * libssh2:x64-windows
  * libwebp:x64-windows
    ogre:x64-windows
  * openjpeg:x64-windows
  * openssl:x64-windows
  * pcre:x64-windows
    protobuf:x64-windows
    qt5:x64-windows
    qwt:x64-windows
  * sqlite3:x64-windows
    tbb:x64-windows
  * tiff:x64-windows
    zeromq:x64-windows
    zlib:x64-windows
    zziplib:x64-windows

As far as I know, even on ROS, most of these libraries are typically obtained from the system package manager (via rosdep), and not compiled through the "tooling for comprehensive source builds", even because several of this libraries have non-standard build system (neither CMake nor autotools). If instead there are tools to compile/obtain easily this kind of libraries in Windows other than vcpkg, I would be happy to try it!

[traversaro]

Thanks for everyone contributing to the discussion!

[podsvirov]

Hello all! The vcpkg version 0.0.91 add support for export to QtIFW based installers. See changelog for more info.
I hope this addition will be useful.

[traversaro]

It will definitely be helpful, thanks a lot for the contribution @podsvirov !

[sivabudh]

So can I install vcpkg through Chocolatey now?

[traversaro]

Hi @sivabudh , sorry but I never noticed your question. I do not think that that is possible, see microsoft/vcpkg#3930 .

[traversaro]

Related to this old discussion, today Microsoft announced the project of porting ROS1 on Windows, https://blogs.windows.com/windowsexperience/2018/09/28/bringing-the-power-of-windows-10-to-the-robot-operating-system/ .

I briefly checked the project, and apparently the ros-win developers (I guess @seanyen-msft and @johnsonshih ) had the same problem discussed in this issue, but they choose to deploy their library (mostly C/C++) binary dependencies as Chocolatey packages, see https://ros-win.visualstudio.com/ros-win/_git/rosdep-au-packages . Those packages seems to just contain the Release binaries. I am not sure how those binaries are generated, but probably they are not generated using vcpkg. Given your great original insightful answer, do you have any opinion on this @ras0219-msft ?

[ras0219-msft]

Unfortunately, MSFT is a big enough place that I haven't talked directly with the team. I will reach out to them though!

[traversaro]

I had missed that, but PR related to the discussion in this issue has been opened against vcpkg back in June: microsoft/vcpkg#6891 .