Addon Manager

[carsakiller]

There has already been some discussion on this topic, but I wanted to create a discussion dedicated to the topic before things get even harder to track there.

Intro

With over 500k installs in VS Code (:tada:), there is a lot of documented Lua out there. People are also creating lots of definition files for providing documentation and completion. To help people find and use these definition files, there should be a more complete solution.

Current Solution

The current method for discovering definition files is just searching around on the internet or by taking a browse through the meta files thread. A thread is definitely not the best medium for finding definition files, though.

Once you find the definition files, you have to download them and manually enable them by specifying their path through workspace.library or workspace.userThirdParty. This isn't terribly hard, but a user can't enable a library if they can't find it, so discovery should be a priority.

Possible Solution

There is a lot that could be done to improve the experience. This is my opinion with some thought put towards it, but I am of course open to feedback, so feel free to join in on the conversation!

Official Definition Repository

There should be a new repository created that stores the definition files of the definitions currently included in the language server. Those files currently included in the source code can then be removed and moved to this new repository. This also has the bonus of letting people contribute to these definition files without opening issues and PRs on the language server repo. This repo can then grow to document all kinds of projects, similar to how DefinitelyTyped works for TypeScript.

Community Definitions

Should someone have definition files that they haven't or maybe don't want to add to the official repository, they should still be able to use them. Users can then manually include these “custom” definitions similarly to how it presently works.

Discovery and Installation of Official Definitions

Using the GitHub API and GitLab API it is possible to programmatically get all the projects in the new official repository.

Now, the language server could download the config.lua (if existing) for each project, but at some point, there will be a lot of projects defined in that repo. I'm not sure how performant it would be to download hundreds of config files and run them all. I instead propose users opt-in to the definitions they want installed. This can be done in VS Code through the extension's webview API, and could look something like this:

As for NeoVim, I have never used it, so I don't know if they have any similar function. We could always create a github.io site that allows users to browse through the available projects in an equally easy-to-read and “pretty” way. They could then download them directly from that site and be prompted with instructions of where the downloaded files should go and what settings they need to modify to enable the definitions.

[vaintrub]

It is great idea😁. How about keeping definitions for libraries in separate repositories and somehow add a link to them in the repository here?

And in the settings when downloading the extension, it will be possible to check the boxes for those libraries that will be needed, and then only they will be downloaded.

This seems to get rid of two problems:

• Problems of storing all definitions in one repository
• Downloading them all at once locally.

We are slowly making a library too and it would be great to embed it natively in the language server. But we will constantly update it, so it's more convenient if we work in our own repository.

[carsakiller]

How about keeping definitions for libraries in separate repositories and somehow add a link to them in the repository here?

I think we could do this:

• Have the official repository that contains all the documentation collaborated on as a community.
  • This allows people to easily work together and quickly get into making changes without tracking down various repositories.
• In the official repository, have a JSON file that contains links to external projects
  • Allows us to define some “non-official” documentation projects.
  • Can be read by the extension (would allow the “non-official” projects to appear in the webview)
  • This file can be more easily updated as it is in its own repo, and we won't need to open a bunch of PRs here on the language server.

And in the settings when downloading the extension, it will be possible to check the boxes for those libraries that will be needed

I'm not sure if I understand what you mean here. What checkboxes when downloading?

[vaintrub]

I meant only the extension settings, maybe it will be convenient to choose which libraries you need and which you don't.

[vaintrub]

By the way, your suggestion seems convenient

[sumneko]

I think it would be easier for people to use and understand if it were defined without the action entry:

This is a good idea, I can implement it.
The current implementation is just due to the inner interfaces.

[sumneko]

I think we should use a JSON file instead of a Lua file for the config.

I think it would be easier for people to use and understand if it were defined without the action entry:

All done.

And thanks to copilot:

[carsakiller]

Awesome! I am going to be very busy over the next few weeks as I am entering finals, but I'll get that schema done as soon as possible 🙂

[carsakiller]

Okay, so I have changed the schema so that now I can use the schema used by VS Code and just extend it to a new schema used for “addons”. I think addons is a good name as it can apply to plugins, definition files, environment emulations, or whatever additional user content we could have.

This results in a config.json that looks like this:

{
  "$schema": "../schemas/addon.schema.json",
  "name": "Example Definition Project",
  "description": "An example configuration file for an addon",
  "filename_pattern": ["example_trigger%.lua"],
  "text_pattern": ["trigger_activation"],
  "settings": {
    "diagnostics": {
      "globals": ["myGlobalOne", "myOtherGlobal"],
      "disable": ["different-requires"]
    },
    "workspace.preloadFileSize": 1000
  }
}

Few things to note:

• The Lua. prefix has been dropped. It makes it impossible for me to re-use the settings schema used by VS Code.
• files is renamed to filename_pattern as it is more representative.
• words is renamed to text_pattern as well for the same reason

[carsakiller]

Alright, I added some READMEs to vscode-lua-webvue to hopefully help explain things.

I have noticed something interesting as well. The GitHub API has a pretty low 60 requests (per 15 minutes?) rate limit, which was clearly too low if you wanted to download multiple addons. So, I implemented GitHub authentication using VS Code so that the limit is greatly increased... but after running some tests, there seems to be some caching or something because I can't really get it to rate limit me. This may mean I don't have to strongly suggest users sign in, which would be nice, but I think this needs more testing.

[sumneko]

Great work!

[carsakiller]

Yet another progress report. I have refined things a lot further, and I think it is ready to integrate.

@sumneko, how should we go about discussing merging it in? It is a pretty big addition and makes a few modifications to the VS Code client such as adding a new setting and refactoring some of the code. I'm not 100% sure how your build and release workflows work, so I could use some help integrating my code so that it can install its dependencies and build correctly without interrupting your workflow.

I did some more testing, and it turns out the GitHub API has a rate limit of 60 requests per hour for unauthenticated traffic. So I spent a bit more time on the sign-in process and making it more clear why it is pretty much mandatory.

[sumneko]

I thought it last night and prefer:

• LuaMao (cat in Chinease)
• LuaNeko (cat in Japanese)
• LuaMiao (cat says in Chinease)
• LuaNyao (cat says in Japanese)

No, most of them are already taken.

[carsakiller]

LuaMeow is available

[carsakiller]

Any updates?

[sumneko]

Maybe just LuaLS ?

I created it first, we have chance to rename it later.

[carsakiller]

Sounds good, thanks! Also, note that https://github.com/sumneko/lua-language-server will continue to forward to the organization repo while a new repo does not exist. https://github.com/orgs/community/discussions/22669#discussioncomment-3237667. I can go through the wiki at some point soon hopefully to fix all the links.

[carsakiller]

The addon manager is great for VS Code users, but what is the plan for other clients? VS Code users will no longer need the addons to be included in meta/3rd, but neovim users won't have the addon manager. They could download the addons they need from LLS-Addons or from LuaCATS (or elsewhere) and do the old workspace.userThirdParty.

Personally, I think libraries should be installed by the user and not just included in the language server. I think the server should just be the server, and libraries/addons should be installed with intent by the user. This prevents bloat and the user having libraries that they never plan to use. I have only ever used the included busted and luassert libraries, but that is easy enough for me to drop into a directory and link to with workspace.userThirdParty. The built-in libraries add a little bit of convenience, while also potentially adding a lot of bloat and unnecessary maintenance in this repo.

Alternatively, there could be a setting that is a list of git remotes that the language server could make sure are installed. This way the addon manager can add addons to the list, other clients can easily add links in there and let the server handle it, and when you share your project with someone, assuming they have the server, it would install the needed libraries.

@sumneko interested to hear your thoughts. Ideally we can find a solution that works for all clients 🙂

[sumneko]

I think the meta/3rd directory should be removed.

In addition, I have the following ideas about the addon manager:

1. Client manager addons and downloading.
2. Client notify server to apply some addon in directory instead of notify setConfig one by one.

In addition, I also considered letting the server manage addons, and the client is only used for display.
But there are some issues:

1. Can server resolve Github Token?
2. Are there any other clients who want to download libraries through other APIs (such as package manager)?

However, we can now enable this feature on VSCode. The ideas I mentioned should not cause any break changes to the existing feature.

[carsakiller]

1. Client manager addons and downloading.

I am wondering if the server should be responsible for downloading the addons though. If the server downloads them, then users that don't even have an addon manager (like neovim users) can potentially still let the server handle all the more tedious work (download, workspace.library, additional settings).

2. Client notify server to apply some addon in directory instead of notify `setConfig` one by one.

Agreed. We could simply use sendRequest like we currently do, but that would only allow the addon manager to talk to the server. To make this work with other clients that don't have an addon manager, we could maybe have a setting, workspace.addons. When the server detects a change in this setting, it can install, apply/unapply.

---

In addition, I also considered letting the server manage addons, and the client is only used for display. But there are some issues:

1. Can server resolve Github Token?

The GitHub API is no longer used. The addon manager currently uses Git to install the addons, which are stored as submodules in LLS-Addons. So theoretically, so long as the user has Git installed, the server could provide the same functionality of installing and uninstalling addons.

2. Are there any other clients who want to download libraries through other APIs (such as package manager)?

That is a good question. Because the addon manager is using Git, it should be able to download any public repository that can be cloned over HTTP (GitHub, GitLab, etc.). If there is a specific package manager API, that would likely not work.

However, we can now enable this feature on VSCode. The ideas I mentioned should not cause any break changes to the existing feature.

Sounds good, just adding a bit more polish before opening the PR 👍

[carsakiller]

Hey @sumneko 👋🙂, could I be added as a contributor (or however that works with the organization) for vscode-lua so I can work on the addon manager easier?

[sumneko]

Done!

[Frityet]

Looking at the progress made, would it also be possible to add support for luarocks, the lua package manager? So the ui would let you add luarocks dependencies, and lls would adjust the path as such @carsakiller

[carsakiller]

I have never used luarocks before, so I don't know what should be added or how it works (especially when it comes to the lua language server).

[Frityet]

I have never used luarocks before, so I don't know what should be added or how it works (especially when it comes to the lua language server).

Luarocks works great with lls,

{

    "Lua.workspace.checkThirdParty": false,
    "diagnostics.groupSeverity": {
        "ambiguity": "Error",
        "await": "Error",
        "codestyle": "Error",
        "duplicate": "Hint",
        "global": "Error",
        "luadoc": "Error",
        "redefined": "Information",
        "strict": "Error",
        "strong": "Error",
        "type-check": "Error",
        "unbalanced": "Error",
        "unused": "Hint"
    },
    "runtime.path": [
        "src/?.lua",
        "src/?/init.lua",
        "lua_modules/share/lua/5.1/?.lua",
        "lua_modules/share/lua/5.1/?/init.lua"
    ],
    "runtime.pathStrict": true,
    "runtime.version": "LuaJIT",
    "workspace.ignoreDir": [
        ".vscode"
    ],
    "workspace.library": [
        "lua_modules/lib/lua/5.1/",
        "lua_modules/share/lua/5.1/",
    ],
    "workspace.useGitIgnore": false
}

Just works with any dependency I install.

[carsakiller]

But luarocks is for installing functional libraries, not documentation, no? If a project doesn't use LuaCATS, then their code isn't documented in a way that can be used by the server?

Would it not make sense for users to just do exactly as you have done and add lua_modules/lib/lua/5.1/ to their workspace.library setting?

I'm not sure what the addon manager could add here other than somehow getting a listing of rocks that have been published to luarocks and then displaying an installation button that just runs the install command. Maybe a little more convenient, but not worth the effort in my opinion if the alternative is just searching the name of the rock and copying the install command.

---

If we are to take a look at how the JS world handles this, there is npm. npm allows people to install functional code, like luarocks (e.g. npm install websocket), but also definitions (e.g. npm install @types/websocket), for TypeScript. The @types are provided by DefinitelyTyped to cover code that doesn't include types itself. This system is by no means perfect, but it is pretty convenient. Because luarocks already exists for the functional code, the addon manager is intended to fill in the definitions side of things. Hopefully LuaCATS will soon become mature enough and widespread enough for it to be included in most code already, then we wouldn't need the addon manager.