Use post-parse JuliaSyntax.tokenize() API

[c42f]

Going through the parser allows us to be more precise and complete.

For example it automatically:

• Determines when contextual keywords are keywords, vs identifiers. For example, the outer in outer = 1 is an identifier, but a keyword in for outer i = 1:10
• Validates numeric literals (eg, detecting overflow cases like 10e1000 and flagging as errors)
• Splits or combines ambiguous tokens. For example, making the ... in import ...A three separate . tokens.

Also uses is_error to detect error tokens and highlight them with the error color.

Requires JuliaLang/JuliaSyntax.jl#221

As I was writing this, I noticed that the syntax highlighter has some heuristics in it to act a bit like a parser (just a very little bit!). For example, highlighting quoted symbols a different color.

So I'm not sure this is the best approach long term - maybe we should just be using the parsed GreenTree instead for highlighting. It would give us several options for richer highlighting. Also, if using the parser itself, we'd be able to highlight any syntax error ranges (not just token errors).

Regardless of that, though, this is a straightforward modification we can do right away...

Screenshot showing outer and abstract contextual keywords highlighted correctly and some token errors

[c42f]

Note that the tests here won't work until JuliaLang/JuliaSyntax.jl#221 is merged and a new JuliaSyntax release is deployed.

[c42f]

I released JuliaSyntax-0.3.3 to make this work if you can kick off CI again. @KristofferC 0.3.3 contains your precompile changes and the new tokenize() API required by this PR, but not the breaking changes to the tree data structures I've been doing in the last week.

[codecov-commenter]

Codecov Report

Patch coverage: 83.33% and project coverage change: +0.07 🎉

Comparison is base (a7f4ca7) 56.61% compared to head (1cd38d4) 56.69%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #306      +/-   ##
==========================================
+ Coverage   56.61%   56.69%   +0.07%     
==========================================
  Files          15       15              
  Lines        1111     1113       +2     
==========================================
+ Hits          629      631       +2     
  Misses        482      482              

Impacted Files	Coverage Δ
src/MarkdownHighlighter.jl	0.00% <ø> (ø)
src/OhMyREPL.jl	20.75% <ø> (ø)
src/passes/BracketHighlighter.jl	88.63% <ø> (ø)
src/passes/RainbowBrackets.jl	100.00% <ø> (ø)
src/repl.jl	13.06% <0.00%> (ø)
src/passes/SyntaxHighlighter.jl	68.75% <100.00%> (+0.80%)	⬆️
src/repl_pass.jl	61.05% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[KristofferC]

I need to update the colorschemes to show a nice red background on errors like in your screenshot.

[c42f]

Oops, I'd broken the docs build with this. Hopefully that's fixed now.

[c42f]

Cool yes updating the color schemes makes sense. I just have this in my startup.jl at the moment.

            cs = OhMyREPL.Passes.SyntaxHighlighter._create_distinguished()
            OhMyREPL.Passes.SyntaxHighlighter.error!(cs,
                OhMyREPL.Crayon(background = (120,50,50)))
            OhMyREPL.Passes.SyntaxHighlighter.add!("myscheme", cs)

We can maybe do better with the errors if we use the full parser but there's a bit of a balance there in terms of avoiding distracting the user when they haven't finished typing. (Might need some heuristic to not report errors due to incomplete syntax etc.) Something for another time anyway.