Ability to pass additional configuration file to colcon via commandline?

[mikaelarguedas]

I'd like to be able to pass global configuration files to colcon when invoking it.

Currently the only way to do it is to set the environment variable ation COLCON_DEFAULTS_FILE at invocation time. But this override the defaults and doesn't extend them.
Is there a way to extend the default configuration with another configuration file?

[dirk-thomas]

Defaults are only to override the command line defaults. Have you tried passing a custom meta file using --metas?

[mikaelarguedas]

Yes I tried. But my understanding is that metas files need entries for packages and cannot take global settings for a verb and apply then to all packages.

here is the file that I use:

{
    "build": {
        "build-base": "/tmp/bouncy_ws/build_debug_isolated",
        "cmake-args": ["-DBUILD_TESTING=ON -DCMAKE_BUILD_TYPE=Debug"],
        "install-base": "/tmp/bouncy_ws/install_debug_isolated",
        "symlink-install": true
    },
    "test": {
        "executor": "sequential",
        "event-handlers": "console_direct+"
    }
}

[dirk-thomas]

Ok, I understand now that you want to customize the non-package specific arguments.

Is there a way to extend the default configuration with another configuration file?

I am still not sure about what exactly you expect here. In the posted snippet you set a few arguments for some verbs. How would you like these to "extend" the default configuration? E.g. the default for build-base is build and your file sets /tmp/bouncy_ws/build_debug_isolated - how should the extending work instead?

Or is the request specific to list values? Do you want your two CMake arguments be used by default (the "default" default has no CMake args) if the user doesn't pass anything custom. If the user passes custom CMake args you want your configured ones to still be used and concatenated with the user provided arguments?

[mikaelarguedas]

If the user passes custom CMake args you want your configured ones to still be used and concatenated with the user provided arguments?

Yeas, ideally I'd like -DBUILD_TESTING=ON to be in my defaults.yaml. But the -DCMAKE_BUILD_TYPE=XX to be only in the "debug/release configuration file" without having to repeat -DBUILD_TESTING=ON in all my "overlay configuration files".

Ok, I understand now that you want to customize the non-package specific arguments.

Yes that's what I'd like. To use the defaults.yaml for things I always use (symlink-install, -DBUILD_TESTING etc) and provide an additional file to extend the defaults with configuration specific arguments. For example: extending cmake-args with other parameters, overwrite parameters that can have only one value like symlink-install

[dirk-thomas]

Yeas, ideally I'd like -DBUILD_TESTING=ON to be in my defaults.yaml. But the -DCMAKE_BUILD_TYPE=XX to be only in the "debug/release configuration file" without having to repeat -DBUILD_TESTING=ON in all my "overlay configuration files".

I don't see a good way to achieve this.

Yes that's what I'd like. To use the defaults.yaml for things I always use (symlink-install, -DBUILD_TESTING etc) and provide an additional file to extend the defaults with configuration specific arguments. For example: extending cmake-args with other parameters, overwrite parameters that can have only one value like symlink-install

Allowing multiple default files should be easily doable in colcon-defaults. The logic for merging the data already exists for the .meta files.

[dirk-thomas]

Just as a side note: default files passed through command line arguments would not be able to alter command line arguments defaults anymore - since they are only available after parsing the command line arguments. So this idea would probably need to be realized differently.

[KazNX]

+1

I would also like to see switchable, global package configuration on top of the existing package level configuration. The meta file system is great for configuring packages in the current project, but doesn't support global options. Meanwhile the COLCON_DEFAULTS_FILE supports global configurations but cannot be easily switched - I find the environment variable setup too indirect. I would like to have an easy way to switch build profiles and I can see a few ways to do this:

• Add a command line option to select the COLCON_DEFAULTS_FILE instead of using an environment variable
• Extend the meta file system to:
  • Support global build options like those in the COLCON_DEFAULTS_FILE
  • Extend/Merge data with an existing meta file
• Create a profile system akin to catkin's

The either of first two options would create the desired effect, but I would ultimately prefer a profile system.

[dirk-thomas]

@KazNX you might want to check the new package colcon-mixin and its data repository which gets us one step closer to a "profile like" system. The main advantage of the mixins is that multiple can be used together instead of being limited to one at a time. It doesn't have a "default" mixin yet though (like the profile system of catkin_tools does).

[dirk-thomas]

I'd like to be able to pass global configuration files to colcon when invoking it.

@mikaelarguedas Regarding the original question: with the new mixins this seems to be almost doable. Currently all .mixin files need to be somewhere under $COLCON_HOME/mixin which isn't very convenient. Maybe adding another option like --mixin-file where you can pass the path to a specific file would satisfy this use case?

[KazNX]

@dirk-thomas From a cursory review, it looks like the mixin extension could achieve the desired effect. However, I'm not sure it's clear enough that "--mixin" = profile control and it's not a "set and forget" mechanic. On the latter point, getting the command line wrong may result in build failures as in some cases mixing debug/release builds can fail.

[dirk-thomas]

I'm not sure it's clear enough that "--mixin" = profile control and it's not a "set and forget" mechanic.

There certainly needs to be a way - the same as it exists for profiles in e.g. catkin_tools - to pick a specific set of mixins by default. This could be done using a configuration file in the root of the workspace as it is done for .meta files.

getting the command line wrong may result in build failures as in some cases mixing debug/release builds can fail.

The same argument can be made for:

• any command line argument if the user passes contradicting arguments,
• using multiple mixins which are not compatible with each other as well as
• profiles if they contain incompatible options.

So I don't see were mixins are any different.

[mikaelarguedas]

With a combination of defaults and mixins I can now keep my original workflow (as long as it doesn't rely on env var expansion). Thanks @dirk-thomas !

Feel free to close this issue (don't want to cut short the ongoing discussion)

[dirk-thomas]

To summarize the ticket:

• There is only a single COLCON_DEFAULTS_FILE which populate the defaults. Since this happens before command line arguments are parsed a custom file can't be provided via a command line argument (since that would be too late available in the pipeline).
• The .meta files can be used either from a location like $COLCON_HOME/metadata or custom .meta files can be passed on the command line, but these files can only be used to provide package metadata - not command line arguments.
• The .mixin files satisfy the goal of providing additional sets of configuration, but until now only .mixin files available under $COLCON_HOME/mixin were being considered. There was no way to pass a custom .mixin file using the command line.

colcon/colcon-mixin#7 implements the previously suggested option --mixin-files to enable passing custom .mixin files through the command line.

I will go ahead and close this ticket assuming that this new option satisfies the use case. Please feel free to continue commenting.

[paulbovbel]

Is there a ticket open for implementing a profile-like system that's defined and automatically used per-workspace?

[dirk-thomas]

Is there a ticket open for implementing a profile-like system that's defined and automatically used per-workspace?

No, not at the moment.