Ability to create a cache for a different job

[fidsamurai-crelio]

What feature do you want to see added?

We have a use case where we build 15 modules on react, the steps and codebase are the same, only the name of the module changes, we run all 15 in parallel using the build jobs declarative step.

What I'd like to request is the ability to run a "cache generator" pipeline which will use the steps like yarn install to create the cache and then save it for use by another job.

Upstream changes

No response

[jonesbusy]

I have same requirement here. For cache storage optimization and don't multiply identical caches accros several instances

One job is creating cache (it's a main Jenkins instance). Other instances can consume (read-only) cache.

Ideally we should be able to override the

• Prefix added in Add support base prefix #263
• The job name

[repolevedavaj]

I am thinking if it would make sense to introduce a new step or even a new plugin which is able to handle that in a better way. More similar to the Github cache action but more flexibility. The current api gets more and more complicated and having a completely different step in the same plugin is strange imho.. what do you think @jonesbusy ?

[jonesbusy]

Is right that the cache steps is complicated.

It could be a new plugin or new step I don't have strong opinion about that.

The annoying point is that you will ask users to migrate but at least start from a clean codebase that address most issues we have with jobcacher

[repolevedavaj]

I will check how we could achieve that. Maybe there is an elegant way :)

[jonesbusy]

Perhaps, I think we are not far to use jobcacher at large scale. My pain point is now that I need to optimize the storage.

I did it already per job using skipSave and skipRestore parameter depending on the branch, but now I would like to do it across job and even instances (by setting a read-only ACL on specific folders of the S3 storage)

[repolevedavaj]

A solution to the initial problem here is to allow storing a cache instance-wide and then add this location as an additional candidate when restoring. I assume this will not solve your issue @jonesbusy as you want to use a cache across multiple instances. I don't know whether it makes sense to handle that in the plugin. Maybe we could allow specifying a global readonly cache item storage?

[jonesbusy]

Yes that's correct. Overidding the prefix and job name is not enough. We need to restore or save on different location.

So perhaps an optional global configuration for a general read only cache could work.