[Enhancement] Support for out-of-tree modules

[Flow-It]

Maybe this is already possible, but I haven't found any documentation or other statements. Coming close is issue #4533, but that developed into a different direction.

I propose to allow users to implement their own modules (inheriting ExtensionModule) and allow to import them locally.

Meson already provides a variety of modules for common tasks (i18n for example) or library-specific things (Gnome module for example). There are some issues with in-tree modules though:

• The process of getting a module upstreamed takes a long time if it happens at all (only modules useful for a broader audience are upstreamed).
• One might not be so eager to update the build system (meson itself) just to use a newly added module.

There are already ways to implement custom tasks in meson, mainly find_program() or generator(). Those generalized functions can, by design, not be as convinient to use as specialized functions. The ability to define custom methods with a speaking name and clearly defined parameters makes working with common tasks much more pleasent.

The reasoning behind this request is based on the anforemented issues, as well as companies having internal or proprietary tooling for which a module could legally not be added to meson.

[jpakkane]

This is extremely unlikely to happen. In addition to the same philosophical reasons as for not having user-defined functions in the scripting language there are two additional hurdles:

1. We would need to define an actual API for modules.
2. We would need to keep it stable over releases.

The first one we don't have and we don't have resources for the latter. Not to mention that it would severely limit any internal refactoring we'd need to do to update the core.

[xclaesse]

Technically there is nothing that really prevent you from doing it, I have seen a project doing that already. But this is certainly not something we want to officially support in the near future, or probably ever.

Some reasons from the top of my head:

• We would have to design and maintain a stable API for modules. I recently started to add an API, but we are far from being able to commit to stability. IMHO we are unlikely to ever have the resources to maintain that.
• Meson has the "batteries included" philosophy. That means we rather prefer people to contribute and maintain their modules upstream where it can benefit everyone.
• Allowing 3rd party code means half broken code that gets copy/pasted everywhere and never really gets polished. We all have seen the horror m4 macros did to autotools.
• We want to hear about real world use-cases and see how we can provide proper solutions, rather than everyone doing its own thing in a crappy module. It happens many times in the past that "just a simple helper" turns out into a fully fleshed feature that benefit much more use-cases than anticipated.

[Flow-It]

I can't really agree with some points mentioned:

In addition to the same philosophical reasons as for not having user-defined functions

Then what are the in-tree modules? The ability to "extend" Meson with turing-complete scripts already exists. But I understand there is concern to what extent the language itself would have to change for such a feature. That would entirely depend on a hypothetical modules API.

Meson has the "batteries included" philosophy. That means we rather prefer people to contribute and maintain their modules upstream where it can benefit everyone.

And I'm sure that wouldn't change. We see that in lots of other open source projects. If there is benefit for many, people are willing to upstream their work and contribute. The proposed external modules would fill the gap where a) contributing has legal/confidentiality hurdles and b) upstream would reject such additions as they are too specific (though very useful to the individual).

Allowing 3rd party code means half broken code that gets copy/pasted everywhere and never really gets polished. We all have seen the horror m4 macros did to autotools.

How is that different from the current situation, for example with find_program? The reference manual encourages the use of external scripts for more complicated tasks. Meson is in no way responsible for the quality of those scripts people use and share. I do not know how it came to that "horror", but I assume it was either the inability or the unwillingness of autotools to include comonly useful features upstream to encourage people to migrate away from their half-baked hacks. From what I've experienced Meson is different in this regard.

We want to hear about real world use-cases and see how we can provide proper solutions, rather than everyone doing its own thing in a crappy module. It happens many times in the past that "just a simple helper" turns out into a fully fleshed feature that benefit much more use-cases than anticipated.

See my response to "batteries included" argument. IMHO there is not really a technical solution to this, only encouraging open discussions (and I very much value how the Meson project communicates). I feel that people are voicing their wishes and problems with the software they are using, including build systems like Meson.

That being said, there are no counter arguments against the 2 most important issues you mentioned: A stable modules API and the need for maintenance. I really hope though we could get there some day in the future :)

[eli-schwartz]

The problem with user defined functions or user defined modules both, is that they allow and de facto encourage people to write turing-complete build files. Allowing user defined modules but forbidding user defined functions just means that instead of adding def func() to meson.build you add it to custom_module.py and add import('custom_module').

Requiring the module to be upstreamed in meson means:

• it can be vetted to ensure it doesn't expose something like mod.eval('python code')
• everyone operates on the same playbook, all documentation required to understand the build description can be found in the canonical meson docs, you don't need to read the source code of custom_module.py

Find_program is very different, since it has limited interaction with meson.build and meson is still fully in control of how build targets are declared, added to build.ninja, installed with meson install, etc... and the find_program can only implement the compiler to use.

[eli-schwartz]

I don't have an answer to the topic of "contributing a module has legal/confidentiality hurdles".

[Flow-It]

Thanks, I didn't know about the limitations of find_program(). That clears up the concerns @jpakkane has about UDFs.

[eli-schwartz]

2. We would need to keep it stable over releases.

The first one we don't have and we don't have resources for the latter. Not to mention that it would severely limit any internal refactoring we'd need to do to update the core.

Actually IMO this is not necessarily a problem (if it were the only argument against them) :D we could do like the linux kernel and say "your punishment for not upstreaming it is, there is no stable API and it might break in a point release, you'll need to maintain ifdef soup to keep it working".

[nils-werner]

I would like to add that I believe that Meson could really benefit a lot from a plugin system.

I can see that a lot of the design of Meson is quite elegant and coherent, and very elegantly solves a lot of build system issues that people have with other systems. I believe it is simply the most elegant build system, even for extremely complex C/C++ projects.

However, when you start to approach the edges of Mesons elegant and coherent design space, and need to resort to custom tricks in order to accommodate for another (granted, badly designed and annoying) third-party build tool, the experience deteriorates quickly.

I believe because of this, over time there started to appear a quite a few points of contention in this project, especially in these corner cases where Meson's design and those of some third party tool started to conflict, e.g. #2173 #2320 #2617 and probably many more.

In these cases there started to appear a lot of unsolvable disagreement between users and maintainers, or maintainers among each others, and discussions often came to a standstill. It seems that in many of these cases, people keep on voicing their discontent, which of course is very unpleasant and draining for the maintainers. And maintainers who don't necessarily have the energy to argue, come by every now and then just to tell people that they're doing it wrong, or that they don't agree with people's ideas.

This lead to e.g. #6485 where @xclaesse suggested to delegate modules to different maintainers, in order to take some of the work off of @jpakkane's shoulders. Part of the proposal was that these maintainers were allowed to make independent decisions, and work on their individual design ideas, however @jpakkane understandably wanted to keep the power for final decisions, which again lead to a standstill.

Or to tickets like this one here.

I absolutely see the danger of people creating their own Python scripts just to create what you call "turing complete build scripts". I am absolutely aware of the legacy of those horrid setup.py scripts where people thought they were smart, but that only made your eyes bleed. I am a big proponent of declarative build systems, and I am not proposing to allow people to write their own Python scripts to do whatever.

What I am instead proposing is something like the Pytest Plugin ecosystem. Pytest is an awesome Python test framework that is incredibly elegantly written and extremely flexible and powerful. But its developers recognized that they cannot solve and maintain code for every single usecase, idea, or third-party tool out there, so they created a plugin system instead.

In addition to interfaces to other tools. what this plugin ecosystem also created is a plurality of ideas and freedom for experimentation. Because not every idea needs to be stabilized and upstreamed, people became free to experiment. They can try different ideas and approaches, make mistakes, and learn from their mistakes. Instead of a whole build system breaking, you'd only have small plugins breaking.

Of course you'd introduce version dependencies between plugins and Meson, but that's not the problem of Meson, but of each individual plugin. If a plugin is poorly maintained or stops working, it and its ideas will eventually die out. If however a plugin is nicely maintained and popular, it and its ideas will survive, and people can learn from it, recognize what directions to take next, and what ideas to upstream.

And of course you'll have that one smart ass who creates meson-turing just to be able to create and run "turing complete scripts". But if people then decide to use it and fall on their faces, they'd have to cry on that persons issue tracker instead of coming here. And ideally slowly people will start to recognize that you were right all along, and you can say "I told you so".

All of this goes against the current monolithic and "batteries included" design philosophy of Meson, but I believe that it's exactly this philosophy that's the reason for all the disagreement and standstill in these uncomfortable edge-cases: Meson is basically a one person show, it's monolithic, it tries to solve everything, and tries to enforce best-practices where it can. But that also means it is a lot of work, it doesn't allow itself to make mistakes, and instead of learning, is much more likely to get stuck.

[jpakkane]

A lot of the issues raised here seem to stem from a basic assumption that a plugin system would reduce the workload of Meson devs. I'd like to contest that. Adding a plugin system means a lot more work for us and, even worse, for things that we have no control over. For example:

Of course you'd introduce version dependencies between plugins and Meson, but that's not the problem of Meson, but of each individual plugin.

This is not true. Whenever a Meson updates breaks any plugin, we get the blame. This is in fact already happening. See for example this Debian issue. Basically what happened is that rustc has some bug where it fails to link on some platforms and not others. This was filed as an RC but against Meson, because we are the only package in the entire distro to have a test for this. Ergo, the bug was "our fault" (Rust developers basically ignore everything except Cargo, so no help there). I had to spend hours upon hours of work to replicate the issue using only direct calls to gcc and rustc. With plugins this would be happening even more often than currently (which is already annoyingly often).

This is emblematic of any extension method. Even if there were only issues in "downstream plugins", a fraction of those would be filed against Meson. Going through those issues, triaging them, submitting them to their proper place and possibly having to do battle with the plugin author on which one is actually at fault and all that is a burden on us. All of that takes valuable time away from improving Meson itself.

Meson is basically a one person show

That is not true, and is actually kind of insulting towards all the people who do a lot of hard work on the project. In fact most "work" on Meson has been done by people other than me for several years now. Should I get overrun by a bus today, the project could continue just fine tomorrow. Meson is a "one-man show" only in the sense that a 18th century warship is a "one-man show". Yes, there is single captain in charge. Yes, he does give orders. Yes, sometimes the decisions are not popular among the crew, but that is unavoidable when dealing with a large group of people with highly differing opinions an priorities. But said captain does not work alone, making the ship do things at his whimsy (allusions to James T. Kirk notwithstanding), he is kept on a fairly tight leash by two things. Firstly, he has a duty to the "greater good" of his home country. Secondly, if he fails to keep his crew sufficiently happy, they will throw him over the side to be eaten by sharks.

it's exactly this philosophy that's the reason for all the disagreement and standstill in these uncomfortable edge-cases:

The obvious follow-up question to this is then: would this new approach make these edge-cases go away? Clearly it would not. Every Meson project would invent their own solution, which would most likely be incompatible with each other and then combining the two projects into one would not work. This leads to A) fragmentation B) bug reports filed against Meson to which we can do nothing.

[nils-werner]

This is not true. Whenever a Meson updates breaks any plugin, we get the blame.

OK, I was oversimplifying it.

I meant when Meson has a "proper stable plugin-API with proper migration/deprecation paths etc.", you could argue it'd be the plugins fault. Of course, implementing and maintaining such an API is a tremendous amount of work, there is no denying that, and I didn't want to make it sound like there isn't any work involved.

But I don't think you have to start there. You could start with a "the plugin-system is only meant for experimental modules, proceed with caution and don't ask us for help" policy and see where it goes.

Or hide the plugin system behind a nightly-switch like Rust does. Or anywhere in between...

Going through those issues, triaging them, submitting them to their proper place and possibly having to do battle with the plugin author on which one is actually at fault and all that is a burden on us. All of that takes valuable time away from improving Meson itself.

That is true, triaging tickets does not come for free. However, for a genuine comparison you need to compare the amount of time spent triaging against the amount of time spent implementing everything yourself. I don't believe it's true that triaging tickets mentioning meson-cargo is more work than implementing meson-cargo yourself.

That is not true, and is actually kind of insulting towards all the people who do a lot of hard work on the project.

OK sorry, I didn't mean to say that all the work comes from one person, that's obviously not true. I meant that most of the decisions seem to come from one person, and they all have to go upstream. That's where your bottleneck is.

A few examples: In #6485 @xclaesse was denied to relieve that bottleneck by delegating design-decisions to multiple people, in #2617 @nirbheek wrote an experimental cargo implementation that got nowhere, in #2320 @eli-schwartz has been waiting months for approval to start implementing a POC, and many more. They all didn't get stuck because people didn't want to do the work or didn't do the work. They all got stuck because of unsolvable disagreements about the design, or because they are waiting for decisions from you.

To go with your metaphor, it seems to me you aren't just commanding the bridge crew, but are approving every course correction, every use of the transporters and every re-calibration the antenna dish yourself. Of course you can run a ship that way, but I don't think you have to.

And, to stretch the metaphor even further, the strength and resiliency of a ship's crew comes from the fact that they exactly don't do that.

The obvious follow-up question to this is then: would this new approach make these edge-cases go away? Clearly it would not.

I don't think it is that clear. I'm not talking about forking Meson in 100 directions, or allowing each project to go wild with custom scripts. I'm talking about experimental new modules, like a custom_target that allows subdirs, or an experimental cargo module. Of course many of the attempts will fail, but some of them may succeed. You may then adopt the successes, and ignore the fails.

And more importantly: you'd have a concrete implementation to discuss about, instead of some abstract discussions about hypotheticals and what-ifs, that eventually lead no-where.

And right now, regarding the tickets mentioned above, it doesn't seem like anyone is taking any new approaches at all, so I am not sure how that will eventually make those edge-cases go away either...

[eli-schwartz]

To go with your metaphor, it seems to me you aren't just commanding the bridge crew, but are approving every course correction, every use of the transporters and every re-calibration the antenna dish yourself. Of course you can run a ship that way, but I don't think you have to.

More accurately, design decisions require more approval than simply implementing new features and fixing bugs.

Most changes to meson aren't ever reviewed by @jpakkane, but things discussed in e.g. design documents, or approaches to mixing multiple build systems, do.

I'll say regarding the cargo discussions that I'm not really interested in meson executing cargo to build things either, and IMO the design discussions that got held up on a consensus, have put us on the road to a more reliable and predictable handling of rustc code including automatically handling crates without running cargo at all.

If you think nothing has been happening there then you just haven't been reading the release notes, I'm afraid. :) meson natively supports `dependency('foo-rs') going and running a subproject foo, that has a Cargo.toml and no meson.build, and generating meson targets to build it as an rlib so you can link it to your own code. There is in-flight or landed work to get those automatically from crates.io, process features as meson subproject options, etc.