Creating custom DSL method

[ivokosir]

I want to create my own DSL function that would be something like a more complex MANY. How would I implement that? This was my simplified attempt (wanted DSL is bodySemicolon), but parser throws NotAllInputParsedException: Redundant input, expecting EOF but found: model when trying to parse "model Name;"

Attempt:

import { EmbeddedActionsParser, GrammarAction } from "chevrotain";

import * as L from "./lexer";

export class DummyParser extends EmbeddedActionsParser {
  constructor() {
    super(L.DummyTokens);
    this.performSelfAnalysis();
  }

  bodySemicolon = this.RULE("bodySemicolon", (action: GrammarAction<unknown>) => {
    this.MANY(() => {
      this.ACTION(action);
      this.CONSUME(L.SemiColon);
    });
  });

  definition = this.RULE("definition", () => {
    const models: unknown[] = [];

    this.SUBRULE(this.bodySemicolon, { ARGS: [() => models.push(this.SUBRULE(this.model))] });

    return { models };
  });

  model = this.RULE("model", () => {
    this.CONSUME(L.Model);
    const name = this.CONSUME(L.Identifier).image;

    return { kind: "model", name };
  });
}

export const parser = new DummyParser();

[msujew]

Hi @ivokosir,

as per the documentation, the ACTION function should be used for semantic actions. Your action however is a syntactic action, since it calls SUBRULE. You simply need to change this.ACTION(action) to action().

A short explanation what's happening here: During the grammar recording phase, the grammar doesn't execute the ACTION, and therefore doesn't now about the SUBRULE call hidden within. Effectively, your grammar is completely empty, since the recording phase doesn't capture the call to model which contains the only CONSUME calls.

[ivokosir]

I should note that, I used ACTION because if I just called function like action() I would get TypeError: action is not a function in grammar recording phase.

[msujew]

Right, I'm not quite sure how SUBRULE deals with the ARGS argument during the recording phase, but I believe it simply skips that. Anyway, I don't think that using the ARGS here is a good solution in any case. Transforming bodySemicolon into a normal class method would allow you to pass callbacks into it while maintaining the modularity that you're probably after.

[ivokosir]

I tried with class function but it is very buggy because of DSL indexes. I can't find a way to track which index needs to be used without causing some bugs depending on context that function is called in (eg. what if bodySemicolon is called twice in one RULE)

Is it so that chevrotain just doesn't solve this problem and the best course of action is just to copy-paste and re-implement the pattern every time I want to use it?

[msujew]

I tried with class function but it is very buggy because of DSL indexes. I can't find a way to track which index needs to be used without causing some bugs depending on context that function is called in (eg. what if bodySemicolon is called twice in one RULE)

Hm, yeah that's an issue. In Langium (my use case for Chevrotain) we also build parsers pretty generically based on arrow functions. But it's probably an overly complex solution for your use case. We keep track of indices using a context object, which gets passed throughout the parser builder.

[bd82]

Hello @ivokosir

I am guessing this is the pattern you want to reuse?

    this.MANY(() => {
      // INJECTED GRAMMAR (single subrule?)
      this.CONSUME(L.SemiColon);
    });

How often does this pattern repeat in your source code? It may not be worth the effort to extract.

Anyhow it seems the basic level support for macros should exist: #1004

• Although this is not documented...

Dynamic rules creation

I'm not sure if macros are even needed for this pattern. Could you dynamically create subrules instead?

// e.g using something like:

  function createManyWithSemiColonGrammar(innerGrammar, name) {
      return this.RULE("name", () => {
          this.MANY(() => {
          innerGrammar()
          this.CONSUME(L.SemiColon);
      });
    }

I am not sure its worth the effort for your example, but perhaps the productive use case is more complex

Macros

Avoiding indices conflicts is indeed a problem with macros.
To help mitigate this the valid values for an index was increased to 256 (0-255).

• https://github.com/Chevrotain/chevrotain/blob/master/packages/chevrotain/src/parse/grammar/keys.ts#L9

Additionally the parsing methods (CONSUME/MANY/...) have a variant which accepts the the index as a parameter, e.g:

• https://chevrotain.io/documentation/10_4_2/classes/CstParser.html#consume
• https://chevrotain.io/documentation/10_4_2/classes/BaseParser.html#many

So you could do a little bit of offsetting in your macros, e.g:

export class DummyParser extends EmbeddedActionsParser {
// ...
    
    macroBodySemiColon(idx, subRuleRef) {
           this.many(idx + 10, () => {
               this.subrule(idx + 10, subRuleRef)
               this.consume(idx + 10, L.SemiColon);
        });
    }
    
    definition = this.RULE("definition", () => {
           macroBodySemiColon(1, this.model)
           macroBodySemiColon(2, this.foo)
           macroBodySemiColon(3, this.bar)
           CONSUME3(L.SemiColon)  // won't conflict with `consume` inside the macros because those consumes use 11/12/13 indices
      });
    
    model = this.RULE("model", () => {
        this.CONSUME(L.Model);
        this.CONSUME(L.Identifier);
      });
}

Perhaps this helps?

[ivokosir]

I thought about + someConstant, it looked a bit hackish, but I see no reason that it wouldn't work on all of my cases.

Actual implementation that I want to have is a bit more complex. Grammar would have the ability to have leading and trailing separators, also you could add as many separators in between children. Separator could optionally be a \n (I'm aware this is a bit more complex). This pattern would be present in ~10 places in grammar, sometimes with ; and sometimes ,. While it is possible to implement it in all of those places, it would be best to be modular so adding features/fixing bugs is in one place.

Example of valid grammar:

;
one
two;
three;; four;
five; six
seven;;

I decided to ignore this feature for now as it is mostly a 'syntax sugar' and solution is a bit more complex then I thought. When I implement it I will be sure to write about the solution here.

[bd82]

When I implement it I will be sure to write about the solution here.

Cool.

The main thing missing for "basic v0.1" macros support are actual documentation.
Perhaps your more advanced scenario could be extracted to a runnable example in the /examples/ folder to assist others in the future.