[question] How to combine a compiler package with a CMake toolchain file?

[slietzau]

Hi guys,

I'm struggling to make conan work with a compiler that also needs a CMake toolchain file to work. To fully understand the problem I'll quickly go over the setup. There are two systems:

• a windows based build machine (Windows 10, conan 1.25.1)
• a custom linux host (custom kernel, gcc 4.8, does not run conan)

That means we are cross-compiling from windows to linux. We use CMake as the build system. CMake needs a toolchain file for the compiler (there is a chicken-egg problem in cmake when compilation needs a sysroot to succeed, which is the case for us).
We want the conan ecosystem to be independent from outside factors such as installed software, so naturally it makes sense to package the compiler with conan. To model the dependency between compiler and toolchain file, it would also make sense to include the CMake toolchain file in the package or model the dependency differently.

It's worth noting that the toolchain file needs to know the location of the compilers in order to configure CMAKE_SYSROOT and all the other paths to the compiler tools (e.g. CMAKE_C_COMPILER). The sysroot path is currently passed as an environment variable to the toolchain file (if there is a better way, please let me know).

From reading the documentation, it looks like the modern way is to use two profiles: win_build and linux_host. One of them should have a build_requires dependency on the compiler package. I tried with linux_host but that didn't work out.

So essentially my question is: How is conan designed to work in that scenario?

• Should the compiler package also ship the toolchain file? If not, how to model the dependency?
• Should I use build_requires('compiler-package') in one of the profiles? which one?

A slightly different question regarding the new experimental toolchain feature: Are there any plans to make this feature work together with "old-school" CMake toolchain files such that no modifications on the client side are required?

Thanks in advance for your help, you're doing a great job with conan!

• I've read the CONTRIBUTING guide.

[memsharded]

From reading the documentation, it looks like the modern way is to use two profiles: win_build and linux_host. One of them should have a build_requires dependency on the compiler package. I tried with linux_host but that didn't work out.

Yes, having 2 profiles is the modern way, was released a few months ago and still experimental, but certainly the way to go.

To summarize, to see if I understood correctly:

• You have some "host" packages, libraries and/or executables you want to run on the linux system.
• You have the cross compiler, that runs in Windows but generates binaries for Linux, and want to put that compiler in a Conan package too. That package will contain the compiler and the sysroot? Only the compiler and the sysroot is elsewhere?
• You have one cmake package too, to be used as a build requires
• The "host" packages, for building, at least with some profiles, will build in Windows to be run on Linux. This will build_require both CMake and the toolchain packages. Both packages will run in the "build" context (Windows, x86_64).
• These "host" packages will define in their build() method that they want to use some toolchain.cmake file, that you can provide. I guess that this toolchain.cmake can be provided included in the toolchain package too

If I understood correctly, this would be a build_require for the "host" context, and as such should be declared in the profile_host profile, together with the cmake package too.

Summoning @jgsogo to check this.

A slightly different question regarding the new experimental toolchain feature: Are there any plans to make this feature work together with "old-school" CMake toolchain files such that no modifications on the client side are required?

Yes, the idea would be that the conan_toolchain.cmake can transitively add other cmake toolchain file. Also the user would have all the power in the toolchain() method, so if necessary, it could directly instruct to use another toolchain file and not use the conan one at all, as the toolchain itself could be completely configurable. That is the intention, but still work to do there.

[jgsogo]

Hi! Yours is one of the use-cases we had in mind when designing this new feature, we still need feedback and more examples to polish it before having the final version. So, thanks a lot for sharing with us your thoughts about it. There are more issues about this topic and I'm trying to link them all to #7120.

---

Disclaimer.- Please, realize this is not a final answer, it is something we are working on right now, any of the following alternatives may change or may stop working, or we can recommend in the future a better way to address this scenario. So, please, approach these solutions with caution and do comment and question if you think there is a better way to do it.

---

I would implement your use case as follows (I'm writing just the draft of the recipes, if you need more details about any of them, please ask):

• You have a graph of dependencies (libraries, applications), you wan to compile for your host machine, for the sake of this example we can think about you having a single conanfile with a library, this would look like any other regular package:

  library.py

  class Library(ConanFile):
     name = "library"
     settings = "os", "arch", "compiler", "build_type"
  
     requires = "zlib/1.2.11", "boost/1.72.0",...
  
     def build(self):
        # TODO: We will come back later to this function.
        pass

• You want your tools to be handled by Conan, let's say you are using a package for CMake (the one from Conan Center should work):

  cmake.py

  class CMake(ConanFile):
     name = "mycmake"
     ...
     def package_info(self):
        # We want to inject this "cmake.exe" to the PATH when building our consumers
        bindir = os.path.join(self.package_folder, "bin")
        self.env_info.PATH.append(bindir)

• Same for your compiler, you want it to provide an executable, a sysroot, the toolchain,...

  compiler.py

  class Compiler(ConanFile):
     name = "mycompiler"
     ...
     def package(self):
        # Package everything your consumers will need
        self.copy("<files-sysroot>", dst="sysroot")
        self.copy("compiler.exe", dst="bin")
        self.copy("toolchain.cmake", dst="res")
  
     def package_info(self):
        # We add the compiler to the path
        bindir = os.path.join(self.package_folder, "bin")
        self.env_info.PATH.append(bindir)

• And in order to orchestrate everything you use Conan with the two-profiles (aka xbuild) feature:

  • you need a profile for your build machine, Windows one. This profile will be applied to packages running in that build machine, in this example they will be compiler.py and cmake.py, if they are to be built, they will use local tools like Visual Studio to compile:

    win_build.profile

    [settings]
    os = Windows
    compiler = Visual Studio
    ...

  • and we need another profile for the packages in the host context, we want these packages to be compiled for Linux and to use our own tools, so we need to declare those tools as build_requires for the host context. We can hardcode these build_requires in the libary.py conanfile, but it is not the best practice, we may change these tools in the future, or we might want to compile natively the libraries,.... so the best approach is to add these build_requires dynamically in the profile. We add them to the linux_host.profile because they are going to be applied to packages in host:

    linux_host.profile

    [settings]
    os = Linux
    compiler = <your-compiler>
    
    [build_requires]
    mycmake/version
    mycompiler/version

• All together, command-line usage should work as follows:

  conan export cmake.py mycmake/version
  conan export compiler.py mycompiler/version
  conan create library.py library/version@ --profile:host=linux_host.profile --profile:build=windows_build.profile --build

---

Ok, now let's talk about the toolchain file.

• The toolchain, together with the sysroot, is packaged with the compiler (it makes sense because we only want to use them if we are build-requiring that compiler).

• The toolchain can find the sysroot, they are packaged together and it can use a path relative to the toolchain file itself. This is the layout of the compiler package:

  <cache>/mycompiler/..../package-id/bin/compiler.exe
  <cache>/mycompiler/..../package-id/res/toolchain.cmake
  <cache>/mycompiler/..../package-id/sysroot/...
  

  I really need to try this with CMake to check if it works:

  toolchain.cmake

  set(PATH_TO_SYSROOT "${CMAKE_CURRENT_SOURCE_DIR}/../sysroot/"

  There are other alternatives:

  • using environment variable from the compiler.py recipe

    compiler.py

    class Compiler(ConanFile):
       ...
       def package_info(self):
          self.env_info.SYSROOT_PATH = os.path.join(self.package_folder, "sysroot")

    set(PATH_TO_SYSROOT $ENV{SYSROOT_PATH})

  • using the install() approach under consideration: Feature/new install folder #6957. In that step we can substitute some placeholders in the toolchain.cmake file to point to the actual package folder in the local machine.

The next challenge is how to use the toolchain file to build the package libary.py. New toolchain feature doesn't allow (yet) to override Conan provided toolchain with a custom one. This is something we need to design and develop, it is the roadmap, but it is not available yet, so we need to use the old approach with the old build helper, we cannot add toolchain attribute or def toolchain(self) method to the library.py conanfile.

Old build helper provides a way to inject a custom toolchain by using the CONAN_CMAKE_TOOLCHAIN_FILE environment variable. We could populate this environment variable in teh compiler.py recipe the same way we could add the SYSROOT_PATH or we have added the path to the executable to the PATH... this could be the only transparent way to inject your compiler.py as a build_require without modifying existing library.py recipe:

compiler.py

class Compiler(ConanFile):
   ...
   def package_info(self):
      self.env_info.CONAN_CMAKE_TOOLCHAIN_FILE = os.path.join(self.package_folder, "res", "toolchain.cmake")

library.py

class Library(ConanFile):
   name = "library"
   settings = "os", "arch", "compiler", "build_type"

   requires = "zlib/1.2.11", "boost/1.72.0",...

   def build(self):
      cmake = CMake(self)
      cmake.configure()
      cmake.build()

Other approaches could be to pass information using user_info (available since Conan 1.27), something along this lines, but it requires library.py to know about the existence of mycompiler in the build context, which IMO is highly undesirable:

compiler.py

class Compiler(ConanFile):
   ...
   def package_info(self):
      self.user_info.TOOLCHAIN = os.path.join(self.package_folder, "res", "toolchain.cmake")

library.py

class Library(ConanFile):
   name = "library"
   settings = "os", "arch", "compiler", "build_type"

   requires = "zlib/1.2.11", "boost/1.72.0",...

   def build(self):
      cmake = CMake(self)
      cmake.definitions["CMAKE_TOOLCHAIN_FILE"] = self.user_info_build["mycompiler"].TOOLCHAIN
      cmake.configure()
      cmake.build()

---

Some warnings about the future related to xbuild feature:

We are not sure if propagating all the environment variables from the build_requires and populating the environment before entering the build() function of any recipe in the host context is the way to go. Some build_requires may override others if they use the same names for the environment variables.

I'm thinking about activating the environment for each build_requires only while the executable provided by the build requires is running, something like wrapping the executable:

cmake

source activate_environment
cmake ---propagate command-line arguments
source deactivate_environment

Adding that cmake wrapper to the PATH should intercept any call to cmake in the CLI and it will use the one provided by the build_requires (with the environment declared in the build_requires).

This change would mean that the approach using the env variable CONAN_CMAKE_TOOLCHAIN_FILE wouldn't work out of the box, because that envvar wouldn't be available in the build() function (not before actually calling cmake).

This is just an example.

---

That's all, sorry for the really long answer, but... work in progress 🚧 ⚠️

[slietzau]

Thank you very much for the detailed replies!

@memsharded Your summary is spot on.
It is worth adding that the "host" packages will be built for a number different host profiles(~10) and only the exotic ones require a cmake toolchain file. Putting any logic about the toolchains into library recipes is very undesirable. So for now we used 'CONAN_CMAKE_TOOLCHAIN_FILE` to tell conan and cmake about the toolchain file.

@jgsogo What you describe is exactly what I was trying to accomplish but I failed at propagating the environment variable to the toolchain file. At least in conan 1.25 the env_info is not propagated properly (this might be due to the concerns you mentioned at the end of your answer).

Your idea to use a path relative to the toolchain file provided the missing link 👍. This CMake snippet solved the "propagating the sysroot to the toolchain file" issue for me, thanks!

get_filename_component(SYSROOT_PATH "${CMAKE_CURRENT_LIST_DIR}" DIRECTORY)

I think the idea of wrapping the executables is very interesting. However making this work properly with CMake toolchain files the way I'm currently using them will be a challenge.

Thanks again for the help, I now have a proof of concept.

[jgsogo]

Thanks!

If you can (and want) to share with us the working POC it would be amazing. Having a look at these examples in the wild helps us a lot when thinking about how to evolve the feature.