[Bug] During link step, attempting to rm nested node_modules causes errors if node_modules is a docker volume or similar

[milesrichardson]

• I'd be willing to implement a fix

Describe the bug

In some cases, when using nodeLinker: node-modules, during the link step, yarn will attempt to delete the entire node_modules directory, instead of only its contents. This causes problems with the common docker pattern of mounting node_modules as a volume, resulting in an error like the following:

➤ YN0000: ┌ Link step
➤ YN0001: │ Error: While removing /src/js/cloud/catalog/node_modules EBUSY: resource busy or locked, rmdir '/src/js/cloud/catalog/node_modules'
➤ YN0000: └ Completed in 13.99s

Note in this case, node_modules is in a subdirectory of a monorepo.

To Reproduce

Sorry, I am unable to provide a minimal reproduction.

Environment if relevant (please complete the following information):

• OS: [e.g. OSX, Linux, Windows, ...] Linux + docker
• Node version [e.g. 8.15.0, 10.15.1, ...] v13.11.0
• Yarn version [e.g. 2.0.0-rc1, ...] 2.0.0-rc.30

Additional context

I found this issue -- #1108 -- which may have fixed it. So I tried upgrading yarn using the following command:

root@a9290f036923:/src/js/cloud# yarn set version latest
➤ YN0000: Selecting the highest release amongst 1.22.4 and 18 others
➤ YN0000: Downloading https://github.com/yarnpkg/yarn/releases/download/v1.22.4/yarn-1.22.4.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-1.22.4.js
➤ YN0000: Done in 1.86s
root@a9290f036923:/src/js/cloud# yarn --version
2.0.0-rc.30

But it still seems to be on 2.0.0-rc.30 after running it. As far as I can tell from the releases page, this version does not include the fix linked above, so I am unable to confirm the fix works as I am unable to upgrade to a version including it.

Maybe this is as simple as upgrading yarn. I've searched the docs and am unclear on the best way to do this, could you please advise if so?

[arcanis]

Maybe this is as simple as upgrading yarn. I've searched the docs and am unclear on the best way to do this, could you please advise if so?

Try running yarn set version from sources

[milesrichardson]

Thanks, I actually was in the midst of trying that just now. I've run yarn set version from sources, but it still seems to be using 2.0.0-rc.30. So I try running it with the fully qualified path installed from source, but then get an error about unsupported option ignoreCwd:

root@a9290f036923:/src/js/cloud# yarn set version from sources              
➤ YN0000: Cloning the remote repository                                     
                                                                                
  $ git init /tmp/yarnpkg-sources                                                                      
  $ git remote add origin https://github.com/yarnpkg/berry.git                                                                                                     
  $ git fetch origin master                                                   
  $ git reset --hard FETCH_HEAD                                                                                
                                                                                                                 
➤ YN0000: Building a fresh bundle                                                                                                                                  
                                                      
✓ Done building the CLI!                                                    
? Bundle path: /tmp/yarnpkg-sources/packages/yarnpkg-cli/bundles/yarn.js                              
? Bundle size: 3.02 MB                                          
    → @yarnpkg/plugin-essentials            
    → @yarnpkg/plugin-compat                                                   
    → @yarnpkg/plugin-dlx                                                     
    → @yarnpkg/plugin-file                                                      
    → @yarnpkg/plugin-git                                                      
    → @yarnpkg/plugin-github                                             
    → @yarnpkg/plugin-http                                            
    → @yarnpkg/plugin-init                                                      
    → @yarnpkg/plugin-link                                                     
    → @yarnpkg/plugin-node-modules                                           
    → @yarnpkg/plugin-npm                                              
    → @yarnpkg/plugin-npm-cli                                               
    → @yarnpkg/plugin-pack                                                    
    → @yarnpkg/plugin-patch                                                 
    → @yarnpkg/plugin-pnp                               
                                                                               
➤ YN0000: Saving the new release in .yarn/releases/yarn-sources.js             
➤ YN0000: Done in 7.3m                                                      

root@a9290f036923:/src/js/cloud# yarn --version            
2.0.0-rc.30                                                                 

root@a9290f036923:/src/js/cloud# node /tmp/yarnpkg-sources/packages/yarnpkg-cli/bundles/yarn.js --version
2.0.0-rc.30

root@a9290f036923:/src/js/cloud# node /tmp/yarnpkg-sources/packages/yarnpkg-cli/bundles/yarn.js install
Usage Error: Unrecognized or legacy configuration settings found: ignoreCwd - run "yarn config -v" to see the list of settings supported in Yarn (in <environment>)                                                                              

Sorry if I'm missing something obvious. Maybe this is the latest verison? But I see a .31 in the releases page 16 days ago, whereas .30 was released 29 days ago.

[milesrichardson]

Ok, I was able to set the version like so:

root@a9290f036923:/src/js/cloud# yarn config set yarnPath /tmp/yarnpkg-sources/packages/yarnpkg-cli/bundles/yarn.js
root@a9290f036923:/src/js/cloud# yarn --version
2.0.0-rc.31.git.20200409.44553f37

and the original bug of this issue persists when running yarn install -- it attempts to delete the entire inner node_modules folder instead of just its contents, resulting in the same error at the link step:

➤ YN0000: ┌ Link step
➤ YN0001: │ Error: While removing /src/js/cloud/catalog/node_modules EBUSY: resource busy or locked, rmdir '/src/js/cloud/catalog/node_modules'
➤ YN0000: └ Completed in 9.54s
➤ YN0000: Failed with errors in 16.09s

[larixer]

@milesrichardson Should be fixed via:
#1185
To try it use:
yarn set version from sources --branch 1185

[milesrichardson]

@larixer Wow, quick fix! Thank you :)

Unfortunately I've already nuked the docker container where the bug was happening and started fresh, and I'm not sure what the exact trigger of it was, so unable to reproduce at this time. But next time I encounter the error I will test this out.

[larixer]

@milesrichardson I think it should reproduce every time when you have empty node_modules directory inside docker container. When you have node_modules/.yarn-state.yml file in docker container, it will not reproduce

[nnennajohn]

@larixer Thanks for the quick update. I found this thread searching for the same problem. I am using this in a Docker file though. How can I know when your branch gets merged in? That is, I currently have it as yarn set version from sources --branch 1185. How would i know when to change it back to yarn set version from sources, without the --branch?

[larixer]

@nnennajohn You can subscribe to updates on the pull request web page, and you will be notified on all the commits, comments and the merge of the pull request. The pull request page is here:
#1185
search for Subscribe button in the right sidebar

[nnennajohn]

@larixer Ok. Will do that. I appreciate the quick response. Thank you.

[larixer]

Fixed via #1185

[milesrichardson]

FYI this has started regressing for me again (yarn tries to remove the entire workspace/node_modules directory instead of just its contents). I'm working around it for now by removing the top level node_modules/.yarn-state.yml in the workspace root.

[larixer]

@milesrichardson It's hard to tell anything without reproduction, preferably the Sherlock reproduction

[milesrichardson]

tbh it's unlikely I will get that as the workaround is fine for now. But I will post it if I have it.

[maysupan]

👌

[larixer]

@milesrichardson Probably that PR might help which might be landed into 3.x, but I'm not sure of ETA or whether it will be landed the same as it is now:
#2487

[ncjones]

I'm trying to switch from yarn1 and I'm running into this error. I have added reproduction steps at https://github.com/ncjones/yarn-nm-volume.

I have not found a workaround yet; removing the .yarn-state.yml does not help.

[larixer]

@ncjones This PR should help:
#3467

yarn set version from sources --branch 3467

[Taraluktus]

I can confirm that #3467 does NOT help in this case (node_modules as docker volumes).
In Yarn 3.1.x, we used the workaround of deleting the .yarn-state.yml before executing yarn install, which worked, but was very annoying.
In 3.2.0 it doesn't work at all, so we had to revert to 3.1.1.

[larixer]

@Taraluktus @ncjones Your issues are unactionable from our side without reproduction. Please open an issue with reproduction.