Parser-based highlighter

[snuyanzin]

Currently org.jline.reader.impl.DefaultParser can answer if cursor is inside a quoted string or a comment.

1. The idea is that during parsing this info could be stored (e.g. in an array, string, etc) and then used in a ParserBasedHighlighter.
2. Some syntax could have keywords which could be provided e.g. via supplier
3. Probably it makes sense to provide several preconfigured color schemes for highlighting
4. API for registering a custom color scheme (perhaps could be a separate task in future)

I would like to contribute here if there no objections

[mattirn]

JLine already has a syntax highlighter Nano.SyntaxHighlighter that can be used to highlight command lines. Also Groovy REPL command line highlighting is implemented using Nano.SyntaxHighlighter.

I guess you are interested in SQL syntax highlighting. Copy sql.nanorc to your git/jline directory before launching Groovy REPL to experiment Nano.SyntaxHighlighter.
Note: I have added highlight rule for block comment highlight in sql.nanorc.

 % ./build repl jansi               
Launching Groovy REPL...
...
groovy-repl> 
groovy-repl> terminal=org.jline.console.SystemRegistry.get().terminal()
groovy-repl> 
groovy-repl> import org.jline.builtins.Nano.SyntaxHighlighter
groovy-repl> highlighter=SyntaxHighlighter.build('file:sql.nanorc')
groovy-repl> 

[snuyanzin]

As far as I know this highlighter uses regexp under the hood without context info ... And this could lead to lots of corner cases e.g. like

groovy-repl> highlighter.reset().highlight("""
dquote     > select id as `'`, name, '123--', cender,
dquote     > '`value with a back tick' as `value`
dquote     > from employee where id > 10 and name = 'M*`'
dquote     > """).println(terminal)

will highlight as

and that is incorrect

And the expected highlighting should look like this (in this dialect there is a double quote instead of back tick and colors are a bit different however the idea is same)

P.S. name is highlighted as in this dialect it is a keyword

[mattirn]

That's true but we cannot use LineReader.parser to find string literals and comments in command line for highlight purposes. The reason is that we can implement a CLI that can handle many languages. As an example we can have CLI that have ordinary commands with dash-dash option convention (--long_option) and sql command that take as an argument any valid sql statement and execute it (\> sql select id, name from table --comment). Of course we cannot add -- as a line comment delimiter to our parser because then our ordinary commands' dash-dash options will be treated as comments by our parser.

We can resolve the problem you have stated above by enhancing SyntaxHighlighter. In a pull request #752 has been added a theme system to SyntaxHighlighter that allows highlight rules to be specified in terms of token type names. We can add a token type group (names that start with $) to configure a parser that is used for highlighting. In nanorc file (ex. groovy.nanorc)

$LINE_COMMENT      //
$BLOCK_COMMENT     /* */
$STRING            " ' ''' """ 

In default.nanorctheme file:

# 
#  parser highlight rules
#
$LINE_COMMENT  COMMENT \n color brightwhite,cyan "FIXME|TODO|XXX"
$BLOCK_COMMENT COMMENT \n color brightwhite,cyan "FIXME|TODO|XXX"
$STRING        STRING

Parser will be created when parsing nanorc file and we find a first occurrence of parser token type name. The consequent occurrences we will further configure the parser, see

jline3/builtins/src/main/java/org/jline/builtins/SyntaxHighlighter.java

Line 329 in bcc7f30

public void parse() throws IOException {

Parser highlight rules will be then applied here

jline3/builtins/src/main/java/org/jline/builtins/SyntaxHighlighter.java

Line 177 in bcc7f30

private AttributedString splitAndHighlight(AttributedString attributedString) {

I let you take a look to the code. We can later define the Parser API.