Language Server: How to react on external changes of the AST

[agonzalezgaliana]

I'm coming from the following SO question

As you can see, our requirement is to be able to model our model instances (only one *.ecore for the whole application) by means of different editors (Theia Tree editor and Textual representation by means of Xtext). Very similar to the Coffee editor, but (unfortunately) not exactly as how the Coffee editor works with the textual representation. As I understand from playing around with the Coffee editor (and by taking a look to the code), the textual modeling in the Coffee editor does not model a coffee machine (Coffee.ecore) but a workflow (Workflow.ecore) that somehow has some dependencies on the machine objects of the Coffee.ecore. The workflow modeling is somehow isolated and only when the *.coffee files are changed in our workspace, the Language Server reloads the coffee models and validates the workflow model against the machines.

In our case, we expect the Language Server (and hence the textual representation) to react on changes done in our model by means of the Theia Tree editor (i.e. the name of a tree node changes -> textual representation is adapted)

So far we're able to listen to model changes in the Language Server (coming from the Model Server), however we don't know how to proceed (mainly because a lack of knowledge in the internal structures of Xtext's LS). Our guess on what to do would be the following:

1. Model Server sends new model to Language Server (via subscription mechanism)
2. The Language Server replaces the AST with the new model
3. Serialize the new AST and cache it into a Document (?)
   At this point the text file (opened by the editor) is not in sync with the internal AST
   4.a Maybe the Language Server forwards the serialization (text representation) to the editor (?)
   4.b Maybe the Language Server overwrites the text file in the workspace (?)

We were wondering if anyone has faced a similar situation so maybe the solution could be shared here with us. We know that maybe the questions are too specific from the Xtext project, but we're also looking for similar experiences inside an EMFCloud modeling solution.

Kind regards.

[vhemery]

It looks like you are facing a "single source of truth" related issue. I strongly suggest you first have a look at Maximilian's talk on https://www.youtube.com/watch?v=yxS-Xsx-4Tw to understand what I am talking about...
Long story short, if you open a model in both the model server and an unrelated LSP server, then expect modifications to work synchronously without any glue code, you're doomed...

The EMF.cloud model server has some mechanics to reload a modified file on the server which should help. See eclipse-emfcloud/emfcloud-modelserver#115

But it is very important that you defined your model modification flow, and which is the source of truth for your model, and listen to any modification at this source from other participants.
So the first step is to take a pen and paper, start defining all your participants (will you edit the model with a text editor? only use LSP and model server? should the LSP be central and the model server read from it ?), and draw arrows to define how you want them to depend on each other. Then you should have a better understanding of who should reload/listen to what. And finally, you can add glue code to refresh what is needed in the participants. (with issue 115, EMF refresh, file listening...)

[agonzalezgaliana]

Hi @vhemery thanks for your answer!

I'm not sure if I understand you when you say

Long story short, if you open a model in both the model server and an unrelated LSP server, then expect modifications to work synchronously without any glue code, you're doomed...

Our idea is somehow "similar" as what is implemented in the Coffee editor but with a big difference which is to model only one model. In case of the Coffee editor, there are several models involved; the tree editor manages *.coffee files and models the coffee machine structure while the textual editor manages *.workflow files and models workflows. (I think) the coffee model influences the workflow but not the other way around. At some point inside the Language Server there is some "glue code" implementation for the validation of the *.workflow files (i.e. to be sure that the name of the coffee machine actually exists in the coffee model) to do so, the Language Server retrieves the coffee models from the Model Server ("foo.workflow" results in retrieving "foo.coffee" from the Model Server).

In our case, we work with a single model that can be edited in 2 editors (tree + forms and a textual editor). Our architecture is pretty similar to the Coffee editor (imagine that we work with *.coffee files and *coffeetext):

• Theia tree editor + JSON forms (able to work with *.coffee files) connected to the Model Server
• Model server
• Textual editor + Language Server (able to work with *.coffeetext files) also connected to the Model Server

The Language Server is not unrelated because is connected with the Model Server. What we're actually trying to clarify is how the "glue" code should look like in our Language Server so we're able to react on changes done by other editors. An with "react" I mean to automatically edit the textual representation (*.coffeetext) when the corresponding *coffee file has been edited. Therefore our Xtext specific questions.

On the other side, I agree with you, we also need to have a clear picture of the connections of all editors and evaluate their behavior on synchronizing the changes.

[vhemery]

If you have a single model, you do not need 2 files.
I think it may be the workflow model in the coffee editor that influenced and confused you. But in this case, these really are two separate models and with different metamodels.

But if you truly have 1 single model that can be opened in 2 editors, then you should have only one file extension and only one single persisted file. This will save you a lot of synchronization nightmare.

Then, if your Language Server takes your model from the Model Server, that should be all the glue code you need.
LSP -> ModelServer -> Json File (or XMI, or Xtext...)
The other option is to persist your model with Xtext, and to use the Xtext Resource as your model in the model server. (with the constraint that model server and LSP must run in a same server instance, not as independant servers)
ModelServer -> Xtext Resource <- LSP

Both are valid options, and it really is a matter of personal preference. It's important to note where each participant takes its information, what are the loaded Resource models, and how they are persisted.

For the first option, the glue code is for having the LSP rely on the model server objects (like in the talk mentioned earlier), then we let the model server handle the persistence.
For the second option, the glue code is customizing the Model Server by using a custom repository manager injected with Guice, which reuse the same XtextResource as the LSP.

I would personnally prefer the second option, since a repository manager with only a single Xtext model resource is very easy to implement (and it would be safer to customize the repository manager anyway in your case... and your language is probably more compact and readable than json...). But once again, that's only a matter of personal preference.

[agonzalezgaliana]

Hi @vhemery we continued working on this topic and we were able to find a possible solution for our needs which I think it matches more or less the second option you mention (with some synchronization).

In our case the Model Server will store the Xtext EMF resources (as source of truth). Changes done in the models stored in the Model Server (done by other editors) will be forwarded to the Language Server as soon as the Language Server requires them (i.e. when the textual editor opens a textual representation). At some point the Language Server will update the textual representation with the latest status of the models stored in the Model Server.

We still work with 2 kinds of files:

1. The file where the Xtext textual representation is defined
2. The XMI serialization of the semantic model

Since you mention "If you have a single model, you do not need 2 files." I'm not sure if this could be our case. We just have one model (*.ecore) but we model that by means of a text file and an XMI file. I don't think we can get rid of the Xtext textual representation file right?

[eneufeld]

@agonzalezgaliana please mark the question as answered if it is resolved for you, thank you!