All information about 'build' and 'host' environment in the 'host' environment

[jgsogo]

Reality bites. We agree that packages should contain binaries only for one architecture, so the proposed model by Conan with the host and the build contexts, theoretically, would be enough for every use case.

But there are out there some packages like flex (westes/flex#78) or build systems like Meson where there is a common practice: to build artifacts for the build context before building the artifacts for the host context (in the same project, one single Meson run). Usually, we refer to them as multi-stage, or they require some bootstrapping,...

One solution would be always to divide the package, but this is not always possible/desired (?). So I'm opening this issue to discuss about this topic. cc/ @thkowa @TheQwertiest @SSE4

---

Where are we now (only new model using two profiles):

• During the build() method the environment is populated with the environment variables defined in the profile:host, and the environment variables coming from the build context (those defined by the packages, and those needed for tools to run: PATH, DYLD_LIBRARY_PATH,...). But in the host context you cannot reproduce the context you have during while building packages in the build context, even though you have access to settings_build, you cannot access the environment that was defined in the profile:build or environment:build.

• The helper self.run("....", run_environment=True) is noop when using two profiles.

I would say that this covers most of the cross-building scenarios out there...

---

flex, cmake, gcc (?),...

These packages have a first stage which is a native compilation and a second stage that compiles the final product. Current Conan model is valid for this scenario, although the implementation I can imagine is not as straightforward as I'd like it to be.

For example, if flex requires stage1flex to run in the build context and the flex is the final library itself (for the host context) something like this would work:

(pseudocode)

class Recipe:
    name = "flex"

    def build_requirements(self):
        # To cross-compile we need 'stage1flex' from the build context. This conditional is also needed to break the infinite loop
        if self.settings != self.settings_build:
            self.build_requires("flex")

    def build(self):
        # First stage can only be compiled in the "build" machine:
        if self.settings == self.settings_build:
            self.run("<compile stage1flex>")

        # In the build context `stage1flex` is available as we have just compile it in the previous line
        # In the host context, `stage1flex` is available via the build_requires (it is in the PATH)
        self.run("<compile flex using stage1flex>")

    def package_info():
        self.cpp_info.libs = ["flex"]

        # Next line will add to the PATH the generated 'stage1flex' (only if native build)
        if self.settings == self.settings_build:
            self.cpp_info.bindirs = ["bin"]

The problem here: depending on the context the package has generated only the library flex or the tool stage1flex as well. This circumstance has to be encoded into the package ID, so we would need an option just for this:

class Recipe:

    options = {"stage1flex": [None, True, False]}
    default_options = {"stage1flex": None}

    def configure():
        # The 'stage1flex' can only be generated if native building
        assert not self.options.stage1flex or self.settings == self.settings_build
        if self.options.stage1flex is None:
            self.options.stage1flex = bool(self.settings == self.settings_build)

    def package_id(self):
        # If the user requests the package with the library only, we can fallback to the package that contains both
        if self.options.stage1flex == False:
            compatible_pkg = self.info.clone()
            compatible_pkg.options.stage1flex = True
            self.compatible_packages.append(compatible_pkg)

One big advantage of this approach is that stage1flex is generated only once if several packages require this same tool.

Meson

The problem with Meson is with monolithic projects where you cannot choose to build only the cross part, every execution needs to compile the native stage and then the cross stage.

Question: Is this a requirement? Can we add an if into the Meson project definition to choose the part to build? It's Python, it should be possible.

If we can choose the stage to compile from the project, then the same approach I presented for the flex recipe should work for any Meson project too: first it will run the native compilation with all the needed settings, then it will run the cross-compilation with the tools from the first stage in the PATH.

If not,

• is this a common practice in Meson? Is it recommended?
• Does it make sense to be forced to compile both stages?
• How should I identify these packages? It has generated binaries for two contexts! Should I add information from the build context to the package generated for the host context? This is against Conan model!
• Can we imagine a workaround? Should Conan add support for these scenarios?

---

I have more questions than answers, and I would need to run some POCs to prove the pseudocode presented here and other alternatives. Any feedback and opinions are more than welcome. This is something to think about, the answer is far in the future.

Thanks!

[TheQwertiest]

Question: Is this a requirement?

Yes, it is.

Can we add an if into the Meson project definition to choose the part to build?

Meson works (roughly) this way:

1. Configure step:
   • build the full dependency graph (including both native and cross targets if any).
   • build a backend specific project (e.g. .sln)
2. Build step: launch backend build utility.

There are no separate stages in meson for build and cross targets. It's all a single monolithic stage.

If you exclude any targets on step 1., then they can't be used on step 2., not as a meson target anyway: user will have to rewrite their project to work with excluded targets as external dependencies (and that will require them to add a lot of boiler-plate code and needless if branches around every such target).

You can't exclude any targets on step 2.: if you build any target, it will also trigger the build of all it's dependencies (of course, you can build the dependee without the depender, but that's irrelevant to this issue).

It's Python, it should be possible.

Meson is written in Python, but it's language is not Python and it's not a Python module. See meson FAQ for more info.

is this a common practice in Meson?

I don't have enough information on this. So, no comments :P

Is it recommended?

This is a standard scenario for meson and is not discouraged or deprecated. It's recommended in the same way as let's say print() in Python - "we have a tool, you can use it".

How should I identify these packages? It has generated binaries for two contexts! Should I add information from the build context to the package generated for the host context? This is against Conan model!

I would say that it's on package maintainer to pack only relevant stuff (so pack only binaries that are intended for the supplied profile and not just every random crap that you've built). I mean it's possible to violate this rule even now, with the current conan approach - nothing prevents the user from stuffing all non-related binaries in the resulting package.

So, meson packages should be treated the same way: "we give you information about the build system and target profile, give us binaries for this profile". To publish tools built with build profile is counter-intuitive and should be heavily discouraged (in a cross scenario that is, building these tools as a separate package is obviously fine).
But yes, meson syntax allows for installation of most targets (be it build or cross), which is possible in most other build systems as well (since they usually allow to do everything, including shooting yourself in the head), so it shouldn't be a factor.

PS: I have to mention that I'm just a casual(?) meson user and not a meson developer =)

[dcbaker]

Upstream meson dev here, @TheQwertiest asked me to chime in.

So, meson packages should be treated the same way: "we give you information about the build system and target profile, give us binaries for this profile". To publish tools built with build profile is counter-intuitive and should be heavily discouraged (in a cross scenario that is, building these tools as a separate package is obviously fine).

This is exactly the case, we support building targets for the build machine specifically to use those targets as part of the compilation process for the host machine. While we don't actively stop anyone from installing build machine targets, that is a really ugly thing to do, and we don't give you any nice helpers for that.

Here is an example of what I mean that I wrote. We compile this transpiler that reads a project specific DSL and spits out C code, that is then compiled for the host machine. The transpiler isn't installed, it's just part of the build process to produce binaries that will be installed.

Hopefully that helps.

[madebr]

Question: Is this a requirement?

Absolutely

Can we add an if into the Meson project definition to choose the part to build? It's Python, it should be possible.

This requires a lot of patching and is maintainer heavy.
This does not scale.

* is this a common practice in Meson? Is it recommended?

Requiring a toolchain for build and host is a common requirement. Not only used for building compiler utils.
A project sometimes has sources to generate another source, using c sources compiled for the build system.
To broaden this discussion, meson allows to use an infinite amount of toolchains in one script.
So I would like to have a possibility to access a build toolchain/host toolchain/... in a conanfile. With the default being the host toolchain.
So some blending of the toolchains is needed.

* Does it make sense to be forced to compile both stages?

This is a flex specific question.
It makes sense because by building stage1flex and flex in one recipe, there is a guarantee that the resulting flex will be built according to the commands in the conanfile.
Also no intrusive flex patches are required.

In this case, it is possible to re-use a flex for the build system to build a flex for the host system.
But this is not always the case.

Yesterday, I took a look at packaging openjdk, and it also requires a build and host compiler.
Splitting that project is not so trivial.

* How should I identify these packages? It has generated binaries for two contexts! Should I add information from the build context to the package generated for the _host_ context? **This is against Conan model!**

See @TheQwertiest : it's up to the packager to not include any artifacts built for the build system in the package. This is also the case now.
Some helper functions would be nice to make it easier to apply the flags of the build context.

* Can we imagine a workaround? Should Conan add support for these scenarios?

The current situation is a workaround and has met its limits.

Is this issue a good place to submit a recipe as an example of usage of a dream crossbuilding conan API?

[TheQwertiest]

@madebr , AFAIK meson allows at most for three toolchains - build, host and target (source: https://mesonbuild.com/Cross-compilation.html). It does allow for machine-file layering though, but that shouldn't change the limit.

[madebr]

@TheQwertiest
That is probably the case.
I'm no meson export either. I think I mixed the toolchains of meson with meson's ability to use multiple compilers

[dcbaker]

Actually, you only get two distinct tool chains, one for the build machine and one for the host machine. The target machine is only relevant when building a compiler, the target is the machine that the built compiler will compile for. ala you can build gcc that runs on aarch64 (host) on x86_64 (build) that creates binaries for riscv (target).

What we do support is the ability to mix all supported languages. You want to build a project with C, C++, D, java, C#, and rust? sure we can do that.

[SSE4]

this is not something new. it's not even specific to meson or CMake or autotools.
it's very common practice to first compile "build" executable(s), then use them to generate some tables, headers or whatever, and then compile "host" libraries.
the real word example is OpenEXR that compiles toFloat executable to generate float.h header file.

I believe such intermediate executable shouldn't affect on package id. for instance, native x86->x86 build, cross build arm->x86 and cross build mips->x86 should results in the identical package (in the ideal world, at least).

in contrast, I guess target is a bit different story, as it applies mostly to cross-compilers, but I believe it should always affect on package id. e.g. x86 GCC with ARM target will contain x86 executables (GCC, G++, etc.) and ARM libraries (libgcc, libatomic, etc.).

[TheQwertiest]

This issue is directly linked to the following feature request: #7091

[Ericson2314]

FYI "target" should probably be removed from meson [target_machine] doesn't influence anything at all except for user-written code that queries it.

There is just build and host

[Cyriuz]

Qt6 is another use case where it is kind of required to bundle both host and build binaries in the cross built package since all the users of Qt also needs its host binaries to be able to cross build. Qt5 actually seems to have done this internally which worked out fine.

I tried putting Qt itself as a build requirement, as in #10332 and #5018 but it becomes extremely messy which options are applied to what, and since the consumer conanfile also needs it as a build req I ended up building 3 versions of qt (default options that applies only to the direct build req etc). If there was a build step that ran on the host when using 2 build profiles I could make the Qt recipe just build itself twice when cross building and then package the tools from the host as well in package().

[memsharded]

Closing as outdated/solved in Conan 2, please create new tickets if necessary against Conan 2.X, thanks for the feedback.