[Feature] Allow private packages to be used as a workspace: dependency.

[lmcarreiro]

• I'd be willing to implement this feature
• [?] This feature can already be implemented through a plugin (Not sure, if it can, I would be happy if someone points the direction)

Describe the user story

About monorepos projects that produce many npm packages, it is covered really well by Yarn Workspaces or others tools like Lerna or Microsoft Rush. But in a single project, when it starts getting too big or creates more than one deployable in the build process, a common solution is to break this big project (that has only a single package.json in its root) into small packages (workspaces), so each one can have its own package.json with its own dependencies.

The term "deployable" here doesn't mean a npm package, it could be a docker image or any other mean to deploy some node.js app to a node.js server.

Some project could have more than one deployable, like a simple node.js job script that runs at a scheduled time every day and a web site with a node.js backend, each one with its own package.json with specific scripts and dependencies for each task, and both would use some common code shared between those two in a third npm package. All those three packages would never need to get published to a npm registry.

The solution should be simple, but I never found it:

• [rush] Let's make publishing easier and more flexible microsoft/rushstack#904
• lerna bootstrap command does not install the local yarn packages in docker container lerna/lerna#2158
• Copy local package files to node_modules instead create symlink lerna/lerna#2381
• stackoverflow 1
• stackoverflow 2
• stackoverflow 3

A simple problem like this should have a simple solution. Today we have a lot of complex solutions and any of them can handle it well.

Describe the solution you'd like

It could be as simple as changing the behavior of the workspace: protocol in a specific case that today generates an error. It would be perfectly backwards compatible.

For development time (using yarn install) it already works perfectly, no need to change anything. I run my build-watch scripts of all projects concurrently and it works fine with the symbolic links created by yarn.

But when I'm building and deploying my app (using yarn install --production), I know that instead of creating the symbolic links, it fetches from npm registry. But the problem is that it tries to do that even if the dependency is private, resulting in a "package not found" error.

The correct behavior would be something like, for any dependency using the workspace: protocol, yarn should take a look at the target package locally, if its package.json have a private: true attribute, instead of fetching from the registry, it should just run a yarn pack at the dependency directory (that would probably trigger some build from its prepack script), and use the resulting .tgz instead. And this process should run recursivelly if a workspace: private dependency has other workspace: private dependencies in its package.json.

Describe the drawbacks of your solution

For docker images it is common to copy just the package.json and yarn.lock files first, before run the yarn install --production, and copy the rest of the source code (or already built artifacts, depending on the CI process model) after the dependencies are already restored, to optimize the use of the docker cache and layers.

So maybe the best approach when running yarn install --production with workspace private dependencies would be to look at the package.json of the dependency just to check if it is private, to avoid trying to fetch it from the registry and to read the list of the dependency's dependencies. So it would be missing in the node_modules folder (sorry, I don't know how the PnP works yet) just the private dependencies.

These private dependencies could be built, packed and locally fetched in a later step, something like yarn workspaces fetch that would recursivelly call the yarn pack for each private dependency and fetch using the resulting .tgz.

Describe alternatives you've considered

I've already tried workspaces (in yarn v1), Lerna and Microsoft Rush. They all works well at development time, but for build and deployment, all of them seems to be specific to help publish multiple public npm packages to the npm registry. Neither was made to help breaking a big private project into small private pieces.

Additional context

This would be nice working together with the feature of "focused install" (#489) and would resolve the problem of the issue #936 too.

[Larry1123]

I have been working on a plugin that covers most if not all of what you are talking about.
I am still working on adding new features but it currently will pack any workspace dependency and put it into the yarn cache, so both pnp and nm linker can use it like normal. It does a few other things, but it was made with creating minimal docker deployments as it's main focus.
If you want feel free to check it out https://gitlab.com/Larry1123/yarn-contrib/-/tree/master/packages/plugin-production-install.
If you feel it is missing anything let me know on the yarn discord or make an issue on the project.

[merceyz]

Closing as this seems to be solved by using either of these Yarn plugins https://gitlab.com/Larry1123/yarn-contrib/-/tree/master/packages/plugin-production-install or https://yarn.build/