feat: add support for C++20 modules (#174)

[alfaix]

This PR adds support for C++20 modules.

(Semi-)formal syntax description can be found on cppreference.

While implementing support for modules, I also noticed that class template instantiation, as well as explicit function (or class) templates instantiation declarations (i.e., extern (template_instatiation)) were not parsed correctly. I fixed that in the second commit of three. Please let me know if it would be preferrable to split this in two PRs.

I'm not a tree-sitter expert, I just want to experiment with C++20 modules in my neovim.
Thus, the new rules may have a couple issues. The ones that stick out to me are:

1. The PR introduces a few LR1 conflicts. I'm not sure if they can be avoided by precedence - what I tried failed.
2. optional($.export_declaration) needed to be added to (almost) every top-level declaration. Not sure if there is a more elegant way of doing it, as many of them are not grouped together and introducing such a group now would break old queries. I wanted to keep the export specifier as part of the declaration, so introducing a new export statement felt like a worse solution.

Summary:

• introduce new keywords: module, export, import
• add support for new C++20 statements: module, export, import
• fix template_instantiation to match class instantiation, as well as instantiation declarations
• add the export keyword as an optional part of all exportable declarations and definitions

Resolves: #174

[berchn]

Has anyone had a chance to look at this MR yet?

[alfaix]

I merged the most recent master in case someone does want to take a look at the PR, but the build is failing.
AFAICS, it's failing for the same reason as the master build, and I'm not sure how to fix it.

[XVilka]

@alfaix please rebase, to trigger the updated CI checks.

[alfaix]

Rebased, ran tests, seems to be fine. The commit history is a little bit messy, but assuming you'll squash shouldn't matter.
Let me know if you'd like a separate PR with cleaner history.

[amaanq]

This increases state count too much, I believe this can be refactored and improved

[alfaix]

I don't know much about tree sitter so I don't really understand what increased the state count. Is it the $.export_specifier everywhere or the way module statements are parsed? Could you please give me some pointers/links to understand this better?

If it's the export thing then I guess it can be changed to a single export_statement but that will turn half the code into export statements instead of declarations, and would be annoying to work with, right?

[neg-c]

@alfaix there you go:
https://github.com/tree-sitter/tree-sitter/wiki/Tips-and-Tricks-for-a-grammar-author

[ghost]

Working Draft
Programming Languages — C++

(Generated on 2024-04-22 from the LaTeX sources by cxxdraft-htmlgen. This is not an ISO publication.
For historical versions of the document, see Tim Song's cppwp page.)

Note: this is an early draft. It's known to be incomplet and incorrekt, and it has lots of bad formatting.

Contents
1 Scope [intro.scope]
2 Normative references [intro.refs]
3 Terms and definitions [intro.defs]
4 General principles [intro]
5 Lexical conventions [lex]
6 Basics [basic]
7 Expressions [expr]
8 Statements [stmt.stmt]
9 Declarations [dcl.dcl]
10 Modules [module]
11 Classes [class]
12 Overloading [over]
13 Templates [temp]
13.1 Preamble [temp.pre]
13.2 Template parameters [temp.param]
13.3 Names of template specializations [temp.names]
13.4 Template arguments [temp.arg]
13.4.1 General [temp.arg.general]
13.4.2 Template type arguments [temp.arg.type]
13.4.3 Template non-type arguments [temp.arg.nontype]
13.4.4 Template template arguments [temp.arg.template]
13.5 Template constraints [temp.constr]
13.5.1 General [temp.constr.general]
13.5.2 Constraints [temp.constr.constr]
13.5.2.1 General [temp.constr.constr.general]
13.5.2.2 Logical operations [temp.constr.op]
13.5.2.3 Atomic constraints [temp.constr.atomic]
13.5.3 Constrained declarations [temp.constr.decl]
13.5.4 Constraint normalization [temp.constr.normal]
13.5.5 Partial ordering by constraints [temp.constr.order]
13.6 Type equivalence [temp.type]
13.7 Template declarations [temp.decls]
13.7.1 General [temp.decls.general]
13.7.2 Class templates [temp.class]
13.7.2.1 General [temp.class.general]
13.7.2.2 Member functions of class templates [temp.mem.func]
13.7.2.3 Deduction guides [temp.deduct.guide]
13.7.2.4 Member classes of class templates [temp.mem.class]
13.7.2.5 Static data members of class templates [temp.static]
13.7.2.6 Enumeration members of class templates [temp.mem.enum]
13.7.3 Member templates [temp.mem]
13.7.4 Variadic templates [temp.variadic]
13.7.5 Friends [temp.friend]
13.7.6 Partial specialization [temp.spec.partial]
13.7.6.1 General [temp.spec.partial.general]
13.7.6.2 Matching of partial specializations [temp.spec.partial.match]
13.7.6.3 Partial ordering of partial specializations [temp.spec.partial.order]
13.7.6.4 Members of class template partial specializations [temp.spec.partial.member]
13.7.7 Function templates [temp.fct]
13.7.7.1 General [temp.fct.general]
13.7.7.2 Function template overloading [temp.over.link]
13.7.7.3 Partial ordering of function templates [temp.func.order]
13.7.8 Alias templates [temp.alias]
13.7.9 Concept definitions [temp.concept]
13.8 Name resolution [temp.res]
13.8.1 General [temp.res.general]
13.8.2 Locally declared names [temp.local]
13.8.3 Dependent names [temp.dep]
13.8.3.1 General [temp.dep.general]
13.8.3.2 Dependent types [temp.dep.type]
13.8.3.3 Type-dependent expressions [temp.dep.expr]
13.8.3.4 Value-dependent expressions [temp.dep.constexpr]
13.8.3.5 Dependent template arguments [temp.dep.temp]
13.8.4 Dependent name resolution [temp.dep.res]
13.8.4.1 Point of instantiation [temp.point]
13.8.4.2 Candidate functions [temp.dep.candidate]
13.9 Template instantiation and specialization [temp.spec]
13.9.1 General [temp.spec.general]
13.9.2 Implicit instantiation [temp.inst]
13.9.3 Explicit instantiation [temp.explicit]
13.9.4 Explicit specialization [temp.expl.spec]
13.10 Function template specializations [temp.fct.spec]
13.10.1 General [temp.fct.spec.general]
13.10.2 Explicit template argument specification [temp.arg.explicit]
13.10.3 Template argument deduction [temp.deduct]
13.10.3.1 General [temp.deduct.general]
13.10.3.2 Deducing template arguments from a function call [temp.deduct.call]
13.10.3.3 Deducing template arguments taking the address of a function template [temp.deduct.funcaddr]
13.10.3.4 Deducing conversion function template arguments [temp.deduct.conv]
13.10.3.5 Deducing template arguments during partial ordering [temp.deduct.partial]
13.10.3.6 Deducing template arguments from a type [temp.deduct.type]
13.10.3.7 Deducing template arguments from a function declaration [temp.deduct.decl]
13.10.4 Overload resolution [temp.over]
14 Exception handling [except]
15 Preprocessing directives [cpp]
16 Library introduction [library]
17 Language support library [support]
18 Concepts library [concepts]
19 Diagnostics library [diagnostics]
20 Memory management library [mem]
21 Metaprogramming library [meta]
22 General utilities library [utilities]
23 Strings library [strings]
24 Containers library [containers]
25 Iterators library [iterators]
26 Ranges library [ranges]
27 Algorithms library [algorithms]
28 Numerics library [numerics]
29 Time library [time]
30 Localization library [localization]
31 Input/output library [input.output]
32 Regular expressions library [re]
33 Concurrency support library [thread]
Annex A (informative)
Grammar summary [gram]
Annex B (normative)
Implementation quantities [implimits]
Annex C (informative)
Compatibility [diff]
Annex D (normative)
Compatibility features [depr]
Annex E (informative)
Conformance with UAX #31 [uaxid]
Bibliography
Index
Index of grammar productions
Index of library headers
Index of library names
Index of library concepts
Index of implementation-defined behavior

[Tomcat-42]

Whats the current status on this?

[Tomcat-42]

@amaanq @alfaix

[alfaix]

@Tomcat-42 I just use my branch in my neovim and it works. When I have the time, I can go back to this and work on reducing the state count, but for now have other priorities.

If you have the time and the energy - feel free to take over :)

[Tomcat-42]

Thanks for your attention, @alfaix. As you suggested, I continued your work in #266.

Now, regarding the class template instantiation and explicit template instantiation declaration issues that you found, will you have time to open another PR to address them separately?

[alfaix]

@Tomcat-42 sure, will do