[Feature] Self-contained runtime from workspaces package

[ojkelly]

• I'd be willing to implement this feature
• This feature can already be implemented through a plugin

Describe the user story

I want one command to create a zip file with the code from a workspaces package
and all its dependencies.

The primary reason for me, is deploying to AWS Lambda. But this feature is useful to anyone who wants to compile part of their monorepo and discard the rest. For example deploying into a container.

Describe the solution you'd like

I'm not sure how to implement this, though I can easily see this being served
by a plugin if they are capable of this.

A command such as yarn bundle which takes and argument of the local package to
bundle, or if possible can work it out from the directory you're in.

I'm not concerned about the name, its just the one that kept popping into my head.

Then likely following the pnp setting of the workspace, the required dependencies would be compiled and placed in a temporary folder with the contents of the package (such that they are usuable).

That temporary folder would then be zipped, and saved to a location defined either in package.json or but an argument passed in by the user.

We would also need the ability to exclude files or folders.

yarn bundle --out .dist/bundle.zip --exclude .dist

Describe the drawbacks of your solution

I think this is a mostly best practice approach to this type of bundling, but
it may be more opinionated than I'm aware of.

This is not a feature that would be useful for those developing packages that
are published. That use-case is already well covered, this serves a different purpose.

I think things like postinstall would not work, unless you ran yarn install upon opening the package, which I think would defeat the purpose of this. It should be a zero-install, unzip and run situation. I don't think postinstall should run though, given the target outcome.

Describe alternatives you've considered

In the past, I've used hacky approaches with yarn and npm, that keeps node_modules next to the source. The whole package is then zipped up.

This no longer works with pnp.

Yarn already has yarn pack, this would be similar, but for the use-case of deploying, not publishing.

Additional context

This may not fit exactly into the world of a package manager, but I think it does fit into the world that Yarn is heading too.

There's some existing tickets (#489 and #859) that I think touch on the same issue, and similar solution.

[ojkelly]

I'm happy to work on this, I think I can build on top of #489.

I wanted to capture this in a separate ticket as it's explicitly trying to solve deploying to AWS Lambda. If anyone else has the same use case and would like to help.

[sjmeverett]

I also need to publish lambda functions. My lambdas rely on shared functionality in a sibling workspace package, so I use webpack to bundle the code up. Some packages, e.g. bcrypt, are native and can't be bundled, so I then have a script which finds specific named packages in node_modules, and also their dependencies, and copies it all into my lambda zip. Obviously I can't do this with berry.

I guess the way I would do this with the solution proposed by @ojkelly is would be to make sure the sibling packages are in devDependencies of my lambda package, and the tool would only pack up dependencies.

I also thought about some kind of API that says takes a list of package names and a basepath/source package, and gives you a .pnp.js and a list of paths you need to copy, but perhaps this is overcomplicated.

[ojkelly]

@sjmeverett I've been making good progress on this, and a build command.

Currently copying everything to a temp folder, removing all the un-referenced local packages (relative to the one being bundled), and then removing all devDeps and peerDeps.

Then running yarn install to reduce the size. Have got a simple package bundle down to 8mb.

Just need to work out actually running it in lambda. Probably going to drop an entrypoint.js file at the very top, that exports the file listed in pkg.main.

I think, we'll need a custom lambda runtime though, in order to work with pnp.

Should have something to play with in a few days.

[sjmeverett]

Amazing, let me know if there's anything I can help with!

[ojkelly]

@sjmeverett I've got the beginnings of the plugin here https://github.com/ojkelly/yarn.build/tree/master/packages/plugins/yarn-plugin-build

I noticed a need to be able to build the local packages and their deps, so thats actually what i've spent the most time on.

I've got that working now, so I'm going to come back and get the aws lambda runtime sorted out.

There's a bunch of issues on that repo regarding stuff to do/improvements.

Keen to hear feedback on if it works.

[ojkelly]

This is a look at the build command. I'm working on the readme etc, so theres not many details around. But the intent is basically to build on yarn v2 to add capabilities that bazel and buck have.

[lmcarreiro]

It is not the same thing, but I think I could reuse some of your plugin code if I would try to implement a plugin for #1200 (if it is possible).

About the zip, I don't know AWS lambda, but with docker images I think it is not a good idea, for 2 reasons:

1. The docker image would have a single layer with the application+dependencies. If you have 500 MB of dependencies and 5 MB of application, every time you fix a bug without changing anything in the dependencies, you would have to replace that entire layer that could be the same if it were different layers for dependencies vs application.
2. Some npm packages have a post-install script with side effects specific to the OS where they are running. Someone could have problems if running the yarn bundle on windows and using this zip inside a docker image based on linux.

[ojkelly]

@lmcarreiro the example packages in the repo I linked above are all private: true, and they all depend on each other. I think your problem in #1200 may be solved by the build command I added, but not by the bundle command.

The build command looks for all the local packages and works out a DAG by recursing down each package and finding any dependencies that are also local. After that it creates a build queue, and runs the build script in package.json in each package.

The bundle command depends on build having already been run (which is why I wrote the build command). It then copies just whats needed to a temporary directory and runs yarn install.

So, where yarn itself is perfect for managing external dependencies, build is for local dependencies and bundle is for deployment.

I'm working on much more detailed documentation and configurability.

---

1. Docker
   Yarn is pretty good with pnp in not needing to make changes to your deps, so unless you update them that part shouldn't need to change.

I haven't thought a lot about yarn bundle with docker yet, my need is specific to aws lambda right now. However, I've used plenty of containers in the past and it largely looks like another deploy target.

I think theres a bunch of configuration to add to yarn bundle to suit different deployment targets.

For docker, if your dependencies are all in .yarn then all thats likely to change is whats in packages.

2. post-install - in the build and bundle command neither call post-install and I dont think they should (though if theres a reason for it, it could be done) because thats already taken care of by yarn install.

yarn bundle run on windows would most certainly break on linux (if there were os specific deps). But I don't think thats an issue, in that I would strongly suggest yarn bundle should only be used in production as part of a CI/CD pipeline. Building on your machine, then deploying is too unreliable for production usage.

---

I think your use case may be solved by bundle but with some additional work to properly support docker.

If not, I think this is where yarn's plugin model really suits. In that we can have both plugins targeted at slightly different problems.

[ojkelly]

@sjmeverett I've just updated the repo to add a file entrypoint.js to the bundled zip.

It should setup pnp for you, and then rexport the main file. So I think it will work properly with aws lambda. Will confirm shortly.

@lmcarreiro for docker I was just thinking, if you used build layers you could call yarn bundle then send the bundle.tgz to the next layer, unzip it and your done. If you need postinstalls to run, you could call yarn install there tool. And then set your entrypoint to the entrypoint.js file, or just call yarn node <script>

yarn plugin import https://yarn.build

[kherock]

For anyone still watching this, I've picked up development of this and am maintaining a functional workspace exporter, linked here:
https://github.com/kherock/yarn-plugins/tree/master/packages/plugin-workspaces-export

yarn plugin import https://github.com/kherock/yarn-plugins/releases/download/release/2020-10-14/plugin-workspaces-export.js
yarn workspaces export -h

The approach is to pack the current workspace into a temporary folder and reimplement WorkspaceResolver and WorkspaceFetcher so that on install, workspace: dependencies are packed and stored in the cache folder like regular npm dependencies. The yarn.lock at the top of the project is copied to the generated project and discarded after install.

Since my implementation doesn't rely on anything PnP-specific, I've added a --node-linker option where you can override the generated project's install type. So for example, it's possible to develop with PnP, but if you want to bypass PnP's runtime overhead for your production images, you can a zip a bundle using --node-linker node-modules.

[jtbennett]

@kherock Looks interesting! I'll check it out. In the meantime, this is the approach I'm taking, specifically for TypeScript projects:

https://github.com/jtbennett/create-ts-project/blob/main/packages/ts-project-scripts/src/commands/bundle.ts

In depends on a lot of the other plumbing in that CLI, but in summary, it:

• Runs yarn workspaces focus --production
• Copies the root node_modules to the output dir.
• Copies the build output of the app package.
• Recursively follows references in the app's tsconfig.json to get all the workspaces the app directly and independently depends on, and copies their build output and node_modules into the output dir.

That gives me a single directory to deploy.

It would be great to have all this in a yarn plugin!

[ojkelly]

@kherock that's an interesting approach.

My solution does use pnp.js, so it's nice that there's an option for both (ie, if you want pnp or not).

I do still need to get around to documentation and some polish (2020 right..), but either https://github.com/ojkelly/yarn.build (and then running yarn bundle in a package, or @kherock will work to solve this issue.

[ojkelly]

Just to finish off this ticket, I've found some time to write up some documentation on this plugin. And make a site with a bit more information.

Theres some still to do things in the issues queue, but as it is, it's ready to start being used and tested.

Here's a guide on how to use it owenkelly.com.au/posts/2020/yarn-build

And here's a link to the plugin's site yarn.build