Tact frontend API: Improvements for tooling

[byakuren-hijiri]

There are a few possible ways to improve the Tact frontend API that would be helpful for tooling:

• Improve workflow with {Node,Virtual}FileSystem API. These functions should have improved documentation, e.g. noting that they do not support relative paths. Additionally, we might consider encapsulating them making them private to make users of the Tact API only interact with absolute paths when working with compiler internals.
• Support attributes for identifiers #639
• Save comments to AST or create a separate CST structure. This is crucial for developing a formatter (Tact source code formatter #162), and might be useful for other tools to e.g. suppress warnings or add user-defined invariants.
• Save types to AST or improve an access API to type information (ASTs should support annotations #289).
• Document and refactor the frontend pipeline. Currently, the way the compiler prepares to calling the precompile function is quite convoluted. All the arguments of the functions used before precompile should be documented, we should extract small methods when possible, and document them, to make that API more useful for third-parties.
• Optionally, we should consider introducing mappings from ASTNode ID to ASTNode. This approach makes sense only if necessary, for example, when refining the typechecker. It's a common approach; for instance, rustc internals are implemented this way. For tooling it might be useful to access a node by its ID when navigating the AST structure from other IR entries.
• Add more context to every internal Error to be thrown in compiler internals. This is crucial for debugging third-party tools.
• Add AST iterators that perform functional map and fold over nested nodes of the AST. This will enable API users to inspect the AST in a more convenient way, for example, sorting nodes of a certain type. (Done in PR feat(api): Introduce fold/map functions for AST traversal #368)
• AST Comparison API #686
• Make all the definitions needed to access the AST public. We should make at least ASTStore to be public. Currently, it should be copy-pasted each time a tool is going to parse AST, since getRawAST returns that type.
• Separate the process of parsing from creating the CompilerContext. Some tools might generate AST without parsing, so we won't need to pass mocks in openContext.
• Create named union types for fields of AST nodes (see: Tact frontend API: Improvements for tooling #314 (comment))
• Refactoring: Extract methods from the build function (src/pipeline/build.ts) to make it more modular. We need to separate the build functionality from CLI parsing and use different methods to create context, compile, and precompile. This is important to implement in order to enable third-party tools to hook into the compilation pipeline in the most flexible way. Perhaps, the best way to achieve this functionality is to create the Builder class with public methods defining the pipeline.
• Throw TactCompilationError instead Error when possible #645
• Refactor build to Separate Feature Enabling Function #646
• ILogger interface #648
• Finalize index.ts and export it in package.json #776

These are my points after working with the Tact API for a while. Feel free to discuss and extend them and create separate issues from each checkbox when working on these.

[novusnota]

Save types to AST or improve an access API to type information.

This would be insanely useful in tools like language server and such!

In general, I agree with all those points, but implementation of some may just require us to make our own lexer→parser→semantic analysis pipeline, suitable for compiler AND external tooling depending on various steps in it. I'd say this depends on #286, but I may be wrong and we could pull off such feat without compromising the Ohm's toolkit.

[byakuren-hijiri]

I would also suggest creating named union types for AST entries used in fields. For example:

export type ASTContract = {
    kind: 'def_contract';
    origin: TypeOrigin;
    id: number;
    name: string;
    traits: ASTString[];
    attributes: ASTContractAttribute[];
    declarations: (ASTField | ASTFunction | ASTInitFunction | ASTReceive | ASTConstant)[];
    ref: ASTRef;
};

Could be rewritten as:

export type ASTContractDeclaration = (ASTField | ASTFunction | ASTInitFunction | ASTReceive | ASTConstant);
export type ASTContract = {
    kind: 'def_contract';
    origin: TypeOrigin;
    id: number;
    name: string;
    traits: ASTString[];
    attributes: ASTContractAttribute[];
    declarations: ASTContractDeclaration[];
    ref: ASTRef;
};

This will simplify the life of tooling developers by enabling them to reuse these type definitions from the compiler. Otherwise, I find myself copy-pasting these entries in my projects. Here is an example of a function with such a signature implemented in the static analyzer internals:

function getMethodInfo(
    decl: ASTField | ASTFunction | ASTInitFunction | ASTReceive | ASTConstant,
  ): [string | undefined, FunctionKind | undefined] {

[byakuren-hijiri]

Added three more points:

Add more context to every internal Error to be thrown in compiler internals. This is crucial for debugging third-party tools.
Add AST iterators that perform functional map and fold over nested nodes of the AST. This will enable API users to inspect the AST in a more convenient way, for example, sorting nodes of a certain type.
Add an API that provides equivalence checks between AST nodes of the same type. This is needed, for example, in #335 to implement unit tests.

[byakuren-hijiri]

Added while working on tests for #559:

• Refactoring: Extract methods from the build function (src/pipeline/build.ts) to make it more modular. We need to separate the build functionality from CLI parsing and use different methods to create context, compile, and precompile. This is important to implement in order to enable third-party tools to hook into the compilation pipeline in the most flexible way. Perhaps, the best way to achieve this functionality is to create the Builder class with public methods defining the pipeline.