C++20 modules are in: discussing a sane (experimental) design for Meson #5024
Comments
|
I'm not sure I have a good enough grasp on how C++20 modules are supposed to work, does anyone have a link to a good overview of them? |
|
There isn't one. There are three different implementation that are different. The standardisation committee has promised to create a technical specification on how this "should work" but no-one has to actually follow that, though they are strongly recommended to. |
|
Sigh, I love committee hand waving. Does GCC or Clang have any documents on how their implementation is supposed to work? I'm dreading trying to get this information out of the ICC guys. |
|
Food for thought about how to organize things, this will be a series I guess: https://vector-of-bool.github.io/2019/03/10/modules-1.html |
|
First attemp at module mapping: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1484r1.pdf GNU Make modules support prototype: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1602r0.pdf @jpakkane this seems to be the start of some reasonable mapping and a proof of concept implementation of scanning dependencies? |
|
Hi, some links about modules in GCC and Clang: This will also affect e.g., ninja: It might be worth keeping in mind that e.g., Fortran has had modules (and submodules) for a while. CMake handles those quite well (ninja still needs some patches, I think). |
|
Unless the ninja patches go upstream I don't think we can rely on ninja for this. We'll also have to figure out what XCode and VS are going to do about modules, I don't know if it's better to rely on ninja doing it for us if VS or XCode makes us implement this ourselves. (It doesn't seem unlikely that msbuild will rely on VS explicitly declaring the module relationship in the XML. Meson already handles fortran modules and submodules I think, without using anything from ninja. @scivision has done a lot of work in that area. |
FYI: It's very very likely that those Ninja patches will go upstream soon (I'm planning to review the PR this month) and be released with 1.10. |
|
Hi, I'm the author of the Reddit post linked in the description and the CMake C++20 module support, so I can answer any questions you might have. I have a repository with C++ module use cases that I'd be happy to have meson build support added to.
Does it support generated fortran sources though? That's basically what requires the ninja patches we made. I'll be adding a C++ example for this next week to the above mentioned repository. |
I've gotten verbal confirmation that the major compiler vendors would be fine with providing information specified in this thread (continued here and here). I have the patch for GCC. Clang is next on my list, and it sounds like MSVC will as well. EDG said they'd have to follow whatever the other three do anyways, so we get them and all their backends by their other-compiler emulation. If you have any input on the format specified there, feel free to drop me a line. |
|
any progress ? does the meson team know when we should we expect this feature to be released ? |
|
Just to add another CMake perspective, with details of how CMakes implement Fortran modules and submodules and thoughts how they'd do C++ |
|
I'll also note that the required Ninja features have landed and will be included in the 1.10 release. |
|
Here are my thoughts on supporting C++20 modules in Meson: The main problem that has to be solved with modules is the mapping of module names/identifiers to source files and the resolution of dependencies. There are multiple proposals on how such resolution mechanisms can be implemented. They range from an on-demand compiler, build system IPC mechanism and batch compilation to scanning the source files during the build step. The main problem I see with these proposals is that they all (some more than others) depend on additional functionality in the compiler. In an ideal world, the required feature would be implemented by every compiler with the same logic, same limitations, same JSON format, etc. However, this might not necessarily be the case. As a result, code that compiles perfectly fine with GCC and Clang might fail to compile with ICC. In the (granted unlikely) worst-case scenario, a module handler for every compiler has to be written and maintained. Even if there is an interface that is supported by most compilers, a fallback solution is still required for the remaining compilers. This would also leave us with two module mechanisms to maintain. Additionally, there would also be no guarantee that code that works with the "official" mechanism would also work with the "fallback" mechanism. My solution: Let the build system do the entire dependency management by only providing the "fallback" solution. At least for the initial support until there is a universally accepted solution. Since depending on the compiler in any way doesn't work, both batch compilation and IPC are out. This leaves us with scanning the source code with a builtin/shared tool. I am not going to repeat what is already listed there. I want to argue why I think that this scanner shouldn't be implemented in the compiler. Naturally, this scanning step can not cover all possible macro/module interactions. If such an import statement is encountered where it does not know what to do (unknown macro in This artificially restricts what you can do with C++20 modules (module lookup is implementation-defined anyway, so this should be fine). While this restricts what developers can do initially, it would greatly increase the portabilety of the code since every compiler that supports modules should support explicit module mapping. That being said, there is already a relatively small preprocessor implementation in python, writing one for the scanning tool shouldn't be that hard. So something like this could be supported out of the box: #define FOO fooMod
#define BAR(x) FOO ## : ## x
export module FOO;
import BAR(bar);And expanding the subset of module declarations that are supported later on is always possible. The only real problem is dealing with the automatic Another advantage would be that we have full control over the scanning tool and can remove and add features as we please. With a direct compiler integration, we would have to ask all the compiler vendors to change something. Of course, writing such a scanner wouldn't be trivial to build and would ideally be used by multiple build systems (maybe some Meson CMake cooperation?). I would be personally happy to contribute some code towards this if there is a chance that this approach would be used in Meson. These are my thoughts on this issue, so please correct me if I got anything wrong or missed something. |
|
On August 25, 2019 5:32:19 PM EDT, Daniel Mensinger ***@***.***> wrote:
Here are my thoughts on supporting C++20 modules in Meson:
The main problem that has to be solved with modules is the mapping of
module names/identifiers to source files and the resolution of
dependencies. There are multiple proposals on how such resolution
mechanisms can be implemented. They range from an on-demand compiler,
build system IPC mechanism and [batch
compilation](https://nibblestew.blogspot.com/2019/08/c-modules-with-batch-compiler.html)
to [scanning the source files during the build
step](https://mathstuf.fedorapeople.org/fortran-modules/fortran-modules.html).
I'm the author of the latter.
The main problem I see with these proposals is that they all (some more
than others) depend on additional functionality in the compiler. In an
ideal world, the required feature would be implemented by *every*
compiler with the same logic, same limitations, same JSON format, etc.
However, this might not necessarily be the case. As a result, code that
compiles perfectly fine with GCC and Clang might fail to compile with
ICC. In the (granted unlikely) worst-case scenario, a module handler
for *every* compiler has to be written and maintained.
I have already gotten verbal confirmation that the JSON format is ok with GCC (I have a patch), Clang developers, and MSVC. When I asked EDG developers, they said they have to implement whatever the other three do anyways. I don't think we'll have issues with compiler support (it's really easy anyways compared to having modules at all). Flag spelling may differ, but GCC and Clang will likely be the same.
We at Kitware also have contacts within the Fortran community and may be able to share the format with those compilers too through its standardization process.
Even if there is an interface that is supported by most compilers, a
fallback solution is still required for the remaining compilers. This
would also leave us with two module mechanisms to maintain.
Additionally, there would also be no guarantee that code that works
with the "official" mechanism would also work with the "fallback"
mechanism.
My solution: Let the build system do the entire dependency management
by only providing the "fallback" solution. At least for the initial
support until there is a universally accepted solution.
Since depending on the compiler in any way doesn't work, both batch
compilation and IPC are out. This leaves us with [scanning the source
code](https://mathstuf.fedorapeople.org/fortran-modules/fortran-modules.html)
with a builtin/shared tool. I am not going to repeat what is already
listed there. I want to argue why I think that this scanner shouldn't
be implemented in the compiler.
See clang-scan-deps (on the phone, can get a link tomorrow) for such a tool. Emulating the compilers is hard though, so I think that is probably best at first. Such a tool would be useful for faster scanning if it proves to be too slow.
Naturally, this scanning step can not cover all possible macro/module
interactions. If such an import statement is encountered where it does
not know what to do (unknown macro in `#if`, etc.) an **error** is
produced, and the build is aborted.
This artificially restricts what you can do with C++20 modules (module
lookup is implementation-defined anyway, so this should be fine). While
this restricts what developers can do initially, it would greatly
increase the portabilety of the code since every compiler that supports
modules should support explicit module mapping.
That being said, there is already a relatively [small preprocessor
implementation in python](https://github.com/ned14/pcpp), writing one
for the scanning tool shouldn't be that hard. So something like this
could be supported out of the box:
```cpp
#define FOO fooMod
#define BAR(x) FOO ## : ## x
export module FOO;
import BAR(bar);
```
This is not even the surface of the corner cases :) . Please join #modules and #sg15_tooling on the cpplang slack to discuss with other stakeholders.
And expanding the subset of module declarations that are supported
later on is always possible.
The only real problem is dealing with the automatic `#include`
conversion by the compiler. However, it should be possible to work
around this by having a whitelist of compilers where the rules for
automatic `#include` conversion can be retrieved. For all other
compilers, this feature is disabled.
I'll expand on my thoughts here more tomorrow. Short answer: no conversion at all unless told so by the build system (because it needs to know).
Another advantage would be that we have full control over the scanning
tool and can remove and add features as we please. With a direct
compiler integration, we would have to ask *all* the compiler vendors
to change something.
That's why we're working with ISO where all the implementors are present :) .
Of course, writing such a scanner wouldn't be trivial to build and
would ideally be used by multiple build systems (maybe some Meson CMake
cooperation?). I would be personally happy to contribute some code
towards this if there is a chance that this approach would be used in
Meson.
While I'm not opposed to collaborations, there is effort on such a tool in the Clang world already, so let's collaborate there.
Thanks,
…--Ben
|
I wasn't aware that there is already a project. Less work for us then :)
My idea was specifically not to emulate the compiler, rather provide a tool that works for 90%-99% of use cases and print an error for the rest. Granted, restricting the user is not very user-friendly, but it would guarantee that the code works with every setup.
Sure, my point was (primarily) to give an error and abort the build if such an edge case is discovered. Then support for these edge cases might be added later.
If this format becomes the standard for dependency scanning and every compiler supports it, then this would be the best-case scenario, and my main issue for relying on the compiler is resolved :) In this case, the only remaining issue would be speed. Scanning for dependencies should be nearly instant (I haven't tested any compiler implementation yet, so I don't know about the current performance). It would also be really useful if the compiler would support scanning multiple files at once. This could reduce the process spawning overhead, especially on Windows, |
|
Sorry, got busy and didn't circle back on this. Some links:
Well, the fundamental problem seems to be something like this: #if __has_feature(frobnitz)
import frobnitz;
#else
import fallback.frobnitz;
#endifThe preprocessor definitions are easy to get, but the feature set is not so easy.
Our prior paper (which missed mailings, but is available here) showed how per-source, per-target, and whole-project scanning is possible and isomorphic in terms of build correctness (the difference is mainly in incremental build work reductions). I think |
Even Also, one can work around this in meson (and CMake) by using #pragma once
#define HAS_FEATURE_frobnitz 1Then your original case can be rewritten as: #include "config.h"
#if HAS_FEATURE_frobnitz
import frobnitz;
#else
import fallback.frobnitz;
#endifI will repeat that I am fully aware that this adds additional burden on the user and that not all (presumably valid) use cases can be solved like this one. However, in my opinion, the benefits outweigh the costs here. Such a scanner would be limited by design but blazing fast and easy to maintain. Additionally, I would argue that defining all This scanner would ideally be fairly compact and have no external dependencies. This is because it is even lower on the software stack than build systems like meson and CMake (but not ninja). So only the C++ or python standard library would be allowed as well as a custom That's also why I don't particularly like the idea of a PS: How do you actually use |
The compiler is likely to also be useful as a scanner (GCC is at least). Just probably not as fast as one would like. An option to say "I don't do silly things" to allow using
Well, CMake is unlikely to use modules itself until…oh 2029 probably based on the speed of C++11 adoption of our target compilers, so that part is not a cycle :) .
That, I'm not sure. You'll have to ask Michael Spencer (one of its developers). |
clang-scan-deps is designed for batch processing. It doesn't even have an interface for scanning a single file (other than providing it a batch of a single file). |
clang-scan-deps expects a compilation database. It needs a full command line to know how to preprocess each file.
Upstream clang-scan-deps doesn't currently support reporting modules deps yet. |
|
@jpakkane Ninja v 1.10 seems to have initial support for modules...? ninja-build/ninja#1521 |
|
So, as of today, where do we stand with meson support for C++20 modules? I am willing to ditch CMake for Meson at this point if that support is there. And it is unclear where or how the Ninja support factors in with Meson. CMake supports Fortran, but no indication as far as I know as to when they will port that functionality over to C++20. |
|
The module support is not implemented yet. The compiler toolchains do not support it that well yet either (at least last I looked, maybe it has changed). Once Ninja and toolchains have the necessary bits (and they are sufficiently stable) we'll add module support. |
|
FWIW, ninja does have the necessary bits merged now (and released in 1.10). Compilers still need a spec to write against for dependency info (at least for CMake's approach), but that's mostly on me to work on for SG15. CMake would then need some CMake-side changes for usage requirements related to modules. |
|
This is an implementation of http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1184r1.pdf Module mapper implementation relicensed and freely available: |
|
libcody, build system/compiler communication has been added to GCC. Might be relevant to Meson? |
|
If While that is a solution to building modules, I don't find it a particularly scalable one (you don't know what is needed until the compiler has started and now you're consuming memory, process slots, and doing IPC to figure out what work to do next). It's probably fine for small projects which are writing their makefiles by hand though, but once there's a complicated dependency graph, I only forsee contention. |
Note that I suggest first porting your Fortran scanner to use P1689 as a "our P1689 handling works". It is a future goal to interact with the Fortran standards committee about P1689 as well.
VS has opinions; MSVC goes by the |
|
@dcbaker:
I think the current scanner can also do C++, even if it's fairly primitive still: meson/mesonbuild/scripts/depscan.py Lines 115 to 132 in 6901ec3 I wouldn't mind Meson keeping this direction - I have the impression that this project was always more "practical and pragmatic" than "prepared for all theoretical scenarios". Depending on And especially with https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2019/p1857r1.html, a simple scanner script (maybe running just a preprocessor before it, but maybe not even that) should suffice in most scenarios.
Unfortunately, GCC does not. Here's a snippet of -x language
Specify explicitly the language for the following input files (rather than letting the compiler choose a default based on the file name suffix). This
option applies to all following input files until the next -x option. Possible values for language are:
c c-header cpp-output
c++ c++-header c++-system-header c++-user-header c++-cpp-output
objective-c objective-c-header objective-c-cpp-output
objective-c++ objective-c++-header objective-c++-cpp-output
assembler assembler-with-cpp
ada
d
f77 f77-cpp-input f95 f95-cpp-input
goWhat GCC does have in exchange, is |
|
Oh, I forgot the GCC details…but you can just write a static module mapper; there's no need to do libcody protocol logic at all. It's basically a fancy response file at that point (just without having to deal with command line escaping rules :) ).
There is also P3034R0. However, these only really help with I suspect Meson will more readily be able to do batch scanning than CMake given the lack of per-source flags interfering, but external projects are (almost certainly) going to have crazy logic around |
This is likely future work, the scanner we have seems to be working, and we support a bunch of fortran compilers that are likely never going to grow P1689 support (they're old and not getting updates or have been replaced). All that to say I don't think we'll ever be able to get rid of the hand rolled scanner, but we can probably support P1689 for newer compilers. So it sounds like for gcc their plan is "we will do that right thing depending on what's in the file", clang and msvc give us more power over that? |
|
Aslo, @mathstuf do you know which revision of p1869 the various compilers implement? are the all implementing r5? |
The best practice is to rely on reasonable defaults provided by the compiler. In this case, compiler-provided scanner. This corresponds to the original Meson philosophy set by Jussi. |
Clang and GCC, yes. |
I'm not saying to wait for Fortran compiler support, but to use P1689 for your scanner output and consumption instead of whatever format you use between the scanner and
MSVC is also using r5. |
Ah, that makes sense. Right now our tool just reads fortran files and dumps a dyndep directly. Which works in simple cases, but this reminds me that it's actually broken when using modules across targets... |
|
Yes, I believe it is difficult to have just a scanner that also does the collation step (see this paper that I've probably linked to before in this issue) because the scanner needs to communicate between scans (though one could also have a project-wide scan-and-collate as a single thing, this doesn't work if one supports sources which produce or consume modules generated from tools which produce or consume modules themselves). Target-granularity seems the best middle ground of performance and corner cases if one needs to discover "phases" of targets across the graph at generate time. |
|
Any updates on this? |
|
I have some patches out for Fortran that are groundwork, there are some really annoying to solve because the compiler story is a mess |
|
Small update on the C++ side, module declarations now shouldn't be macros: https://isocpp.org/files/papers/P3034R1.html (Defect Report, so will also apply to C++20 afaik). |
|
Yup: |
|
This was opened in 2019 what is the status? |
|
The current status is that basic cases should mostly work. I have been working on fixing up our scanner and how we do scanning to get faster and more accurate results see here, mainly aiming at Fortran since they've hod modules longer, and C++ and Fortran modules have many of the same design decisions. |
Is there a minimal example somewhere that I can explore? |
|
For C++, in the Meson repo you can look at `test cases/unit/85 cpp modules" This will only work with GCC or MSVC, we don't have support for Clang yet. |
|
Support for clang would be great, as they appear to have the most operational modules. Maybe not a few years ago when the example was done, but now. i.e. https://github.com/infiniflow/infinity/tree/main |
|
So it may be better to mark the status of meson in https://arewemodulesyet.org/tools/ as partial instead of ✅ since it can't work with clang |
|
Hello! |
Project with cpp module, very good reference! I'll try modules from now on Failed to get start with meson .. |
Meson doesn't support this, it would be great if it did (it's CMake). The reason I linked it is that it's a fully working production C++ modules project, so there is no longer the chicken and egg problem. Knowing that a compiler supports them and there is a reference project, it should be easier for Meson to implement with something concrete to look at. (A lot of the discourse around modules seems to boil down to the other guy doesn't support this so how can I. With the other guy being the build system or the compiler, take your pick.) |
|
👀 |
|
For any future travelers...definitely not first class support but I was able to hack together something workable on my project. for a project structure like: meson-native-clang (may not be necessary but been throwing the kitchen sink at this) |
|
Just an FYI, this (using
These are why the build system (IMO) must be involved in hooking up dependencies rather than "just look for things on disk". |
Ah I see. That actually makes sense now why first class support for this stuff is such a big task. Thanks for the education 🙏 |
|
Just to break it down, here's what CMake does (in some more detail than the clouds and boxes in this paper provide:
Complications:
|
and others
Hello everyone.
I am particularly interested in this topic. CMkae guys already started something:https://www.reddit.com/r/cpp/comments/axnwiz/cmake_gcc_module_proofofconcept/
Though I do not know the proposal well enough to propose a particular design myself, I think it would be a good idea to kick off discussion on strategies and module mapping, at least in the context of gcc, now that Modules have been voted in.
I would propose that this thread output is an initial design proposal to give a first try on implementation with the main high level details and strategies:
The text was updated successfully, but these errors were encountered: