Error accessing lookahead token from token stream!

[ryo33]

https://github.com/ryo33/parol_sandbox/tree/d562d4ef567b8b0d91c39f0544e8537c953d4417
This code emits the following error with cargo test

Diagnostic {
  message: "Error accessing lookahead token from token stream!",
  code: "parol_runtime::parser::syntax_error",
  help: "Syntax error in input prevents prediction of next production",
  labels: "[LabeledSpan {
    label: Some(\"Error location\"),
    span: SourceSpan { offset: SourceOffset(0), length: SourceOffset(0) }
  }]"
}

I'm trying to understand what happens, but it's a little harder.

[ryo33]

I replaced .wrap_err("Error accessing lookahead token from token stream!")? with .unwrap(), and it emits Diagnostic { message: "Lookahead exceeds token buffer length" }

[jsinger67]

I will have a closer look at it 😊

[jsinger67]

I guess it's related to the scanner switched back at the very end of the input stream. This really might be a case that is not properly handled yet.
Thank you for reporting this.

Update: wrong guess. See below.

[ryo33]

The last dbg!(self) in eval:

 self = LookaheadDFA {
    states: [
        None,
        Some(
            1,
        ),
        Some(
            5,
        ),
    ],
    transitions: [
        DFATransition(
            0,
            5,
            2,
        ),
        DFATransition(
            0,
            6,
            1,
        ),
        DFATransition(
            0,
            7,
            1,
        ),
        DFATransition(
            0,
            8,
            1,
        ),
    ],
    k: 1,
}

I replaced " with ! and it emits the same error, so it does not seem to be caused by ".

[ryo33]

I think the second time of dbg!(input) is awkward.

[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:43] &rx.rx = (?P<G1>\r\n|\r|\n)|(?P<G2>[ \t]+)|(?P<G5>")|(?P<G6>a)|(?P<G7>b)|(?P<G8>c)|(?P<G9>.)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:44] &input = "\n        \"abc\""
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "\n",
        ),
        "G1": Some(
            "\n",
        ),
        "G2": None,
        "G5": None,
        "G6": None,
        "G7": None,
        "G8": None,
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "        ",
        ),
        "G1": None,
        "G2": Some(
            "        ",
        ),
        "G5": None,
        "G6": None,
        "G7": None,
        "G8": None,
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "\"",
        ),
        "G1": None,
        "G2": None,
        "G5": Some(
            "\"",
        ),
        "G6": None,
        "G7": None,
        "G8": None,
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "a",
        ),
        "G1": None,
        "G2": None,
        "G5": None,
        "G6": Some(
            "a",
        ),
        "G7": None,
        "G8": None,
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "b",
        ),
        "G1": None,
        "G2": None,
        "G5": None,
        "G6": None,
        "G7": Some(
            "b",
        ),
        "G8": None,
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "c",
        ),
        "G1": None,
        "G2": None,
        "G5": None,
        "G6": None,
        "G7": None,
        "G8": Some(
            "c",
        ),
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "\"",
        ),
        "G1": None,
        "G2": None,
        "G5": Some(
            "\"",
        ),
        "G6": None,
        "G7": None,
        "G8": None,
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:43] &rx.rx = |(?P<G9>.)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:44] &input = "abc\""
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "",
        ),
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "",
        ),
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "",
        ),
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "",
        ),
        "G9": None,
    },
)
[/Users/ryo/Workspace/parol/crates/parol_runtime/src/lexer/token_iter.rs:46] &captured = Captures(
    {
        0: Some(
            "",
        ),
        "G9": None,
    },
)

[ryo33]

I've found that String scanner state causes the behavior.
I changed the grammar to ParolSandbox: /"/^ { /a/ | /b/ | /c/ } /"/^;, and now it works.

[ryo33]

%push(String) uses the following scanner, which seems broken.

/* SCANNER_1: "String" */
const SCANNER_1: (&[&str; 5], &[usize; 0]) = (
    &[
        /*  0 */ UNMATCHABLE_TOKEN,
        /*  1 */ UNMATCHABLE_TOKEN,
        /*  2 */ UNMATCHABLE_TOKEN,
        /*  3 */ UNMATCHABLE_TOKEN,
        /*  4 */ UNMATCHABLE_TOKEN,
    ],
    &[],
);

[ryo33]

Oh, I didn't know about terminal attachments for a scanner state.

I've changed the grammar:

String: /"/^ %push(String) { A | B | C } /"/^ %pop();
A: <String> /a/;
B: <String> /b/;
C: <String> /c/;

The error changes to:

Diagnostic {
  message: "Production prediction failed at state 0",
  code: "parol_runtime::parser::syntax_error",
  help: "Syntax error in input prevents prediction of next production",
  labels: "[LabeledSpan { label: Some(\"Error location\"), span: SourceSpan { offset: SourceOffset(5), length: SourceOffset(1) } }]"
}

[ryo33]

String: /"/^ %push(String) { A | B | C } StringWrapper^ %pop();
A: <String> /a/;
B: <String> /b/;
C: <String> /c/;
StringWrapper: <String> /"/;

This works! 🎉

[jsinger67]

Exactly 💯
In your first version there were no terminals assigned to the scanner state ’String’.
The resulting scanner then can't deliver any valid "scannable" tokens.
This should be detected during the analysis phase.
I will add a proper check for this situation.
Thank you for your analysis!