New Plugin/Package managment system

[engelgabriel]

Telescope is another Meteor application that shares our need for modularity and decided to implement its plugin as Meteor packages. This mean that you can add a plugin by using meteor add and remove it with meteor remove and then you have to rebuilt your application and start it.

Discourse has a similar model of plugins as Ruby gems. This is a fine choice for their target audience.

However it is not suitable for us, as we don’t want to force our end users to use a CLI build tool.

The Wordpress publishing platform and the Atom code editor are two good counter-examples that fit in the “modular end-user software” category. They are good examples that matches our goals:

• Plugins can be downloaded from a central repository with an App-store like browsing experience.
• Plugins are hot swappable, which mean that you have neither to rebuild nor to restart the application in order to start or stop a given plugin.
• Plugins can have some dependencies, along with the obvious advantages of an explicit dependencies management, this allows the creation of “packs”.

https://wordpress.org/plugins
https://atom.io/packages

See: https://discuss.wekan.io/t/rfc-the-plugin-system/132

[marceloschmidt]

To accomplish the goal of not having to rebuild it, wouldn't it require that all plugins are already available, compiled, taking up to large amounts of megabytes?

[engelgabriel]

no @marceloschmidt

[Sing-Li]

I think it is fair to say - together with #1485 which has been sitting around with no response since Nov of last year - that the Rocket.Chat community by and large do not care too much about how we organize our code internally and/or how we deal with build system idiosyncrasies.

I can certainly sympathize, and don't blame them, with all the other "more important" new features and issues that scream out for attention.

We may as well forge ahead and create this - since we WILL need it.

[engelgabriel]

@Sing-Li I think that #2246 show that some care.

[ywecur]

There are many features that could be created, but many are not desired by all. The best way to make this a flexible and universal chat solution is to introduce features as packages, with server maintainers themselves choosing which to include. Examples of packages would include things like XMPP support and conversation threading, but would also extend to most features of Rocket.Chat (e.g. Video Conferencing).

This method is currently used by, for example, the text editor Atom through the Atom Package Manager, and with great success. The modular design of the project (a minimal core with packages for specific features) is perhaps its greatest feature and probably the reason it's as successful and universally loved as it is. Disagreements about what to include into Atom are basically non-existent as people could just release a feature as a package rather than having to rely on a majority of users being interested in a feature.

The greatest feature in my opinion, though, is the ability for server maintainers to implement their own features without having to fork the project completely; They could simply write a package that they install on their instance. This would greatly increase the flexibility and usability of the project and lead to a superior number of available features for Rocket.Chat as opposed to its alternatives, probably making it the truly greatest alternative.

[engelgabriel]

And there is this #1741 thread getting to a similar topic. Maybe we should also merge.

[ywecur]

@Sing-Li I can only speak for myself and my server, but we are great supporters of this and would like to help in any way possible!

[jclabaugh]

This would be huge for us in the Enterprise.

[mccambridge]

+1 on this guy. Until this happens, is there any other way to extend functionality besides hacking on the source and working with the API/webhooks?

[engelgabriel]

@mccambridge you can create packages and install them without hacking on the source. This way you wont need to merge to keep updated. This is how most of customisations have been done.

[ywecur]

@engelgabriel To what extent can these packages modify the core?

[engelgabriel]

Most of Rocket.Chat is built with packages, they can access all internal methods and APIs. We are build the system using packages as building blocks exactly so you should be able to do everything you need with packages.

[ywecur]

@engelgabriel Wow, that's amazing that you architected the software in such a modular way right from the get go! Astonishing work!

[timkinnane]

+1 and watching this one intently. I'm working on my first package now, but I don't think it's suitable for core. I can easily add it in my own fork, but I'd love to make it available for others, without necessarily going through a PR to core. As far as I gather, the best way of adding external pacakges currently is to clone them beside your rocket.chat dir, then symlinking them in to the packages, so you're only modifying one file in git?

I've worked with Wordpress plugins a lot and think its absolutely crucial for growing a platform to have a well documented and easy to implement external module manager. Atom modules and the Slack app directory are good examples too obviously, but I think the sheer number of Wordpress plugins and the time they've had to mature their solution makes it a good example to follow.

Some critical things they've dealt with that probably warrant discussion here?

• Module version management, to allow updates to fire a different process to installing, so plugins can handle their own data migrations etc. This extends to what version range of Rocket.chat they're compatible with, to prevent users from installing plugins using unsupported / depreciated functions.
• Enable / disable plugins, for testing functionality or conflicts, without requiring full removal and loss of data.
• An official register of plugins, which probably requires a review team from core to pass judgement and issue criteria for acceptable plugin design patterns, behaviour.

Lastly, another option to consider for the meantime, would be either a yeoman generator or custom CLI tool like https://github.com/vuejs/vue-cli which builds the package scaffold from template repos (I like the way they provide a few approaches for simple and advances use cases). This way you could encourage best practice approaches consistent with core development.

[timkinnane]

Could this be a temporary solution, better than symlinking package dirs?https://www.npmjs.com/package/mgp

Seems like using mgp each instance could have its own git-packages file (git ignored), allowing admins to add their public (or private) packages without modifying core - other than packages file, which is fairly easy to merge.

EDIT: Also experimenting with PACKAGE_DIRS env variable, to have an additional package folder with external repos beside rocket.chat install. This could be easier than above, if anyones following this, let me know if you find a better solution.

[mquandalle]

Does someone currently work on that? Just in case, I also want to point to the VSCode plugin system that I have found to be one of the most polished of my findings: http://code.visualstudio.com/docs/extensionAPI/overview. I think this system has all we need: the ability to define client and/or server services, entry points, semVer enforcement, build scripts, etc. I believe its code could be extracted from the VSCode open-source code base so that we could use it in either Rocket.Chat, Telescope, or Wekan.

[timkinnane]

@mquandalle as I understand it the problem is mostly about how to manage packages from runtime, not in source. Meteor has pretty good built-in modularity. The problem (again, as I understand it) is for instance admins who aren't coders, or even coders who want a more efficient system, there's no easy way to toggle packages without maintaining forks and rebuilding from source.

[engelgabriel]

@mquandalle there is no one working on this yet. We want to start using Meteor 1.4 and mode more packages into NPM before we start looking into how to load packages at run time. Do you have any ideas about this?

[mquandalle]

The simplest solution is to restart the server and reload the client and rely on standard import statements inside conditionals.

If the goal is to be able to do some hot loading (you install a plugin and the UI gets modified 1 second after you clicked install) it will require more plumbing, and especially with Meteor merging everything into one big bundle by default. The idea would be to have different bundles (cf. Webpack) that the client could fetch as static resources. For the client, I remember looking at http://jspm.io/ and it has some machinery to require remote files in a clean way. On the server we would need to have entry points (basically files or functions) that gets called on some event to initiate whatever the server-side package is doing.

At least in the beginning it will be easier to restart the application when a new plugin is installed like what Discourse, Wordpress, Atom, or VSCode are doing. But I’m also sure that with enough engineering effort the plugin hot loading ideal is reachable.

[jykae]

We were also talking today if it would be possible to "hot swap" plugins. Wondering how hard it would be to do so in Meteor? If taken the first approach by @mquandalle how to programmatically I can ask Meteor application to restart when plugin is enabled?

[graywolf336]

See: #6890

[mnlbox]

@engelgabriel any news about this?

[graywolf336]

@mnlbox see my comment about the related issue.

[geekgonecrazy]

@engelgabriel @graywolf336 what do you guys think? Should we go ahead and close in favor of #6890 since that is our chosen route to solve this problem here?

[graywolf336]

@geekgonecrazy yes I agree. Closing this in favor of #6890.