Cache cannot rebuid properly when nested paths specified in mount

[dmitweb]

Describe the bug
I try cache yarn.lock file, but when it's extracted from tar it md5 checksum different. This leads to reinstall node_modules dependencies.

To Reproduce
Steps to reproduce the behavior:

1. Check md5sum before cache and after extract.

Expected behavior
md5sum file before and after must match

Maintainer's note:

Rebuild step fails to find files when nested path given:
packages/test/node_modules

[kakkoyun]

@dmitweb Thanks for the feedback.

Have you actually checked the checksum of this manually by comparing them before and after? I just want to be sure that yarn or npm doesn't intercept in anyhow.

It'd be great if you can provide a simple case to reproduce this, maybe something public?

[dmitweb]

@kakkoyun

Have you actually checked the checksum of this manually by comparing them before and after? I just want to be sure that yarn or npm doesn't intercept in anyhow.

Yes, i checked them. First screenshot from reproduce, second screenshot from my app. Code that check checksum is same. After many tests, I realized that the main problem with caching nested paths. Below i provide repo reproduce issue.

Normal case: after first run drone pipeline, result must be cached and first yarn install must echo
success Already up-to-date.

Repo: after first run drone pipeline, yarn install command reinstall deps

ps. if comment line packages/test/node_modules from mount (restore, rebuild) all works fine, but in my project i have much many deps in package.json, and some of them install in packages/test/node_modules, so i must cache this folder too, this leads to diff in checksums, btw i don't know why

https://github.com/dmitweb/drone-cache-bug

I hope you can figure it out

[kakkoyun]

@dmitweb Thank you very much for your elaborate response. I'll look into it and do my best to figure it out.

[kakkoyun]

The problem still continues.

Logs from with the latest version:

[copy:0] + cp yarn.lock .yarn.lock
[restore:0] level=warn name=drone-cache ts=2020-04-02T20:08:43.579990373Z caller=backend.go:69 component=plugin msg="using aws s3 as backend"
[restore:1] level=info name=drone-cache ts=2020-04-02T20:08:43.58008726Z caller=restorer.go:36 component=plugin component=restorer msg="restoring  cache"
[restore:2] level=info name=drone-cache ts=2020-04-02T20:08:43.580103056Z caller=restorer.go:53 component=plugin component=restorer msg="restoring directory" local=node_modules remote=eb0a191797624dd3a48fa681d3061212/node_modules
[restore:3] level=info name=drone-cache ts=2020-04-02T20:08:43.58013344Z caller=restorer.go:53 component=plugin component=restorer msg="restoring directory" local=packages/test/node_modules remote=eb0a191797624dd3a48fa681d3061212/packages/test/node_modules
[restore:4] level=info name=drone-cache ts=2020-04-02T20:08:43.580189705Z caller=restorer.go:53 component=plugin component=restorer msg="restoring directory" local=yarn.lock remote=eb0a191797624dd3a48fa681d3061212/yarn.lock
[restore:5] level=info name=drone-cache ts=2020-04-02T20:08:43.580301468Z caller=restorer.go:94 component=plugin component=restorer msg="extracting archived directory" remote=eb0a191797624dd3a48fa681d3061212/yarn.lock local=yarn.lock
[restore:6] level=info name=drone-cache ts=2020-04-02T20:08:43.580240156Z caller=restorer.go:94 component=plugin component=restorer msg="extracting archived directory" remote=eb0a191797624dd3a48fa681d3061212/node_modules local=node_modules
[restore:7] level=info name=drone-cache ts=2020-04-02T20:08:43.580372314Z caller=restorer.go:85 component=plugin component=restorer msg="downloading archived directory" remote=eb0a191797624dd3a48fa681d3061212/yarn.lock local=yarn.lock
[restore:8] level=info name=drone-cache ts=2020-04-02T20:08:43.580387349Z caller=restorer.go:85 component=plugin component=restorer msg="downloading archived directory" remote=eb0a191797624dd3a48fa681d3061212/node_modules local=node_modules
[restore:9] level=info name=drone-cache ts=2020-04-02T20:08:43.580423677Z caller=restorer.go:94 component=plugin component=restorer msg="extracting archived directory" remote=eb0a191797624dd3a48fa681d3061212/packages/test/node_modules local=packages/test/node_modules
[restore:10] level=info name=drone-cache ts=2020-04-02T20:08:43.580438461Z caller=restorer.go:85 component=plugin component=restorer msg="downloading archived directory" remote=eb0a191797624dd3a48fa681d3061212/packages/test/node_modules local=packages/test/node_modules
[restore:11] level=info name=drone-cache ts=2020-04-02T20:08:43.580469023Z caller=restorer.go:72 component=plugin component=restorer msg="cache restored" took=371.195µs
[md5:0] + md5sum .yarn.lock
[md5:1] 953488eddea1a7d57fd6124559f14956  .yarn.lock
[md5:2] + md5sum yarn.lock
[md5:3] 953488eddea1a7d57fd6124559f14956  yarn.lock
[md5:4] + yarn install --production
[md5:5] yarn install v1.19.1
[md5:6] [1/4] Resolving packages...
[md5:7] success Already up-to-date.
[md5:8] Done in 0.17s.
[md5:9] + md5sum yarn.lock
[md5:10] 953488eddea1a7d57fd6124559f14956  yarn.lock
[md5:11] + yarn install --production
[md5:12] yarn install v1.19.1
[md5:13] [1/4] Resolving packages...
[md5:14] success Already up-to-date.
[md5:15] Done in 0.28s.
[rebuild:0] level=warn name=drone-cache ts=2020-04-02T20:08:45.648431858Z caller=backend.go:69 component=plugin msg="using aws s3 as backend"
[rebuild:1] level=info name=drone-cache ts=2020-04-02T20:08:45.648517752Z caller=rebuilder.go:39 component=plugin component=rebuilder msg="rebuilding cache"
[rebuild:2] level=info name=drone-cache ts=2020-04-02T20:08:45.649977593Z caller=rebuilder.go:60 component=plugin component=rebuilder msg="rebuilding cache for directory" local=node_modules remote=eb0a191797624dd3a48fa681d3061212/node_modules
[rebuild:3] level=info name=drone-cache ts=2020-04-02T20:08:45.651125807Z caller=rebuilder.go:111 component=plugin component=rebuilder msg="uploading archived directory" local=/drone/src/node_modules remote=eb0a191797624dd3a48fa681d3061212/node_modules
[rebuild:4] level=info name=drone-cache ts=2020-04-02T20:08:45.651316827Z caller=rebuilder.go:99 component=plugin component=rebuilder msg="archiving directory" src=/drone/src/node_modules
[rebuild:5] level=error name=drone-cache ts=2020-04-02T20:08:45.661649292Z caller=main.go:563 err="[IMPORTANT] build cache, source <packages/test/node_modules>, make sure file or directory exists and readable lstat packages/test/node_modules: no such file or directory\n"

[kakkoyun]

@dmitweb I know it's been a while, finally, I've got a fix candidate dmitweb/drone-cache-bug#1

[kakkoyun]

Feel free to open this one again if the latest version doesn't solve your problem. It'll be in the next release otherwise.