[WIP] Vsix packaging - Generate a .vsix for all built-ins

[marcdumais-work]

(eventually) fixes #5

I was able to generate a .vsix file that contains all the built-ins. I had to do a few things:

vscode-builtin-extensions/package.json

• vsce was complaining about some missing fields. Added them.
• vsce does not like having VS Code extensions depend on external things - they have to be self-contained. So I had to remove dependency to @theia/plugin-ext-vscode
• added vsce as a dev-dependency
• added a package script that calls vsce package and saves the .vsix one folder above (so it's easier to find)

Example applications:

• added the @theia/plugin-ext-vscode dependency, removed above, in each, so they can build/work

Note: the example applications does not consume the .vsix at this time, but rather still use "vscode-builtin-extensions": "0.0.0"

TODO: I am still not sure how to publish the .vsix. One option is to use a npm module that makes it easier to publish towards GH releases.

[marcdumais-work]

Note: the generated .vsix does not work at this time. The generated extension manifest does not have entries for the various built-ins, which it probably not correct (missing configurations in this PR, probably).

As well I think we do not yet support extension packs in Theia: eclipse-theia/theia#6611

[vince-fugnitto]

Note: the generated .vsix does not work at this time. The generated extension manifest does not have entries for the various built-ins, which it probably not correct (missing configurations in this PR, probably).

In the package.json, we will likely need to list all available bunded extensions (all builtins), so it can be unpacked by Theia once eclipse-theia/theia#6611 is supported.

For example, in the java-extension-pack the following is described:

  "extensionPack": [
    "redhat.java",
    "vscjava.vscode-java-debug",
    "vscjava.vscode-java-test",
    "vscjava.vscode-maven",
    "vscjava.vscode-java-dependency",
    "VisualStudioExptTeam.vscodeintellicode"
  ],

[vince-fugnitto]

There might be an issue using the builtins as an extension pack as their name are not valid to VS Code. There is some validation which complains about the format of the name. For example, the extensionPack entries follow the pattern:

`${publisher}.${name}

In the case of @theia/vscode-builtin-git:

  "name": "@theia/vscode-builtin-git",
  "displayName": "%displayName%",
  "description": "%description%",
  "publisher": "vscode",
  "version": "0.2.1",
  ...

The following is not supported: (vscode.@theia/vscode-builtin-git):

[marcdumais-work]

 "extensionPack": [
    "redhat.java",
    "vscjava.vscode-java-debug",
    "vscjava.vscode-java-test",
    "vscjava.vscode-maven",
    "vscjava.vscode-java-dependency",
    "VisualStudioExptTeam.vscodeintellicode"
  ],

I tried to add such a block in our package.json, but it looks like the extensions that are defined as being part of the pack are not expected to be bundled in the .vsix but, I surmise, fetched from the marketplace, when the pack is loaded. So this does not work.

So we may need to wait until we have an extension registry that we are allowed to use, and have published there the built-in extensions we build from this repo.

[akosyakov]

It would be nice to publish each built-in extension individually, for it we have to iterate over vscode-builtin-extensions/extensions and run vsce package for each. For typescript we have to copy content of vscode/extensions/node_modules to vscode-builtin-extensions/extensions/typescript-language-features/node_modules to distribute default ts compiler properly.

For now doing GitHub release would be enough. I don't think we can push them to VS Code marketplace. Later we can publish them to our registry then it is deployed somewhere. cc @spoenemann

We should also clean up scripts that there is no anymore renaming of packages, archiving of node_modules, publishing to npmjs and so on.

In Theia repo it is better to consume each extension individually that it is not magic and everybody see the full list of links.

[akosyakov]

If we get rid of publishing to npmjs, how we will publish nightly versions for testing?

@spoenemann would the registry support tags similar to npm? How does it decide what is the latest?

[marcdumais-work]

If we get rid of publishing to npmjs, how we will publish nightly versions for testing?

Good question. I found this related vscode issue:
"Allow extensions to publish beta releases and users to opt-in to them"
microsoft/vscode#15756

I read quickly and it sounds like what the Microsoft marketplace accepts as valid version identifier for its extensions is quite limited and does not permit letters, just numbers. One suggested way forward is to publish 2 distinct extensions: one "stable" and one "preview/beta/nightly".

I am not sure if our own registry will follow the same or maybe be more flexible?

[spoenemann]

would the registry support tags similar to npm?

No. The VS Code marketplace has a notion of tags, but rather in the sense of improving search for marketplace users, not in the sense of version tags. We reuse the same concept in Open VSX.

How does it decide what is the latest?

Using semantic versioning rules, currently implemented in SemanticVersion.java

I am not sure if our own registry will follow the same or maybe be more flexible?

We will also support the preview flag (see extension manifest documentation). But we haven't implemented any constraints on the version string yet. I'd say we're free to be more flexible there, though we should consider the implications for users.

By the way, if you would like to see a preview of our registry:

[akosyakov]

@spoenemann It is more that we need to publish nightly next version which won't change latest version. Would you apply the semantic versioning somehow to decide what is the latest?
i.e. latest is 0.2.1, but nightly is 0.3.0.next.commitid. I guess semver will tell that 0.3.0.next is greater than 0.2.1 and it won't help.

[spoenemann]

You could mark the next version as preview. But currently the marketplace and the extensions view in the IDE don't have any concept of handling multiple versions. They only allow to display and install the latest version, whatever that is.

We could improve this by allowing to browse all available versions like npm does. The REST API already supports that, it's just not covered by the UI.

[marcdumais-work]

closing this - we can make an extension pack for all built-ins when we have our upcoming public registry.