Semantic Coloring

[dvc94ch]

This would be useful for embedded dsls. Thoughts?

[svenefftinge]

Not clear what you mean could you elaborate?

[dvc94ch]

Here is an example of a dsl implemented with racket. Dsl's implemented in racket can register a color-lexer and indenter that is used by Dr. Racket - the official racket IDE. Just finished adding support for formatting and rangeFormatting to the racket langserver. Next goal is to get syntax highlighting and formatting working for arbitrary racket dsls. As you can see on the right the jsonic comment isn't highligthed even dough the langserver should know it's a comment. So I'm suggesting that there be a default tokenizer for the racket language and when the list of document symbols is returned which identifies the kind of symbol, we re-color the symbols more accurately (based on the information the registered color lexers pass to the lang-server).

[gorkem]

Isn't this basically semantic highlighting?

[dvc94ch]

Not sure what it has to do with semantics (type checking rules and evaluation rules) but it seems that this is what people call it. So what needs to happen for semantic highlighting? Is this a duplicate of #1683?

[svenefftinge]

Yes, that is semantic coloring. We will soon write a real proposal for that for the LSP and then implement it here and for the TS, Java, Go, Xtext language servers.

[svenefftinge]

Is this a duplicate of #1683?

No, in that ticket we discuss 'syntactic' coloring based on regexps or lexers.

[svenefftinge]

Are you going to publish that Racket extension somewhere? Looks cool :)

[dvc94ch]

Do you want to add another repo to theia-ide? It's probably not too popular out of an academic setting and may have a similar target audience as xtext, although I doubt that any company being able to afford typefox support would consider this an alternative...

[svenefftinge]

Academic is good and choice, too. I don't know how much effort it would be to support it, but if you or someone else is willing to take care of that it would be great. It should be a separate repo (below theia-ide or not).

[dvc94ch]

So what are your thoughts on the protocol?

There are a couple of implementation issues to consider:

1. If the language server is relied on as the primary source of syntax coloring there is a considerable startup delay. This can only be solved in the client, the client should serialize the coloring state and load it when opening a file.
2. When the semantic coloring information is only available when there are no syntax errors. Should this be handled on the client or the server? I can implement some heuristics on the server to reduce that. The current implementation sends the entire coloring state to the client on each change. The question is if the server should send updates to the client and the client tries to do the best with the information it has or the server should do most of the work to keep the client implementation simple.

[svenefftinge]

This is best to be discussed on https://github.com/Microsoft/language-server-protocol.
I once tried to discuss something along those lines microsoft/language-server-protocol#7

[kittaakos]

One note on the Xtext language server support (maybe it is relevant for TS, Java, and Go as well): I assume we would like to support the semantic coloring in the diff editors too. If yes, it could be tricky to get the semantic coloring information for a historical revision. Historical revisions are not part of the workspace anymore.

[kittaakos]

There is another use-case we need to support in Theia: the preview editor. When for instance the user triggers the Find All References from the editor.

[kittaakos]

Here are a few more related links for this task:

• Support semantic highlighting
• How to Contribute to the Language Server Protocol
• Proposal for (Semantic) Coloring (see #18)

Edit: updated the links.

[kittaakos]

microsoft/vscode-languageserver-node#368