Memoization isn't feature complete

[JPZ13]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

This issue is to aggregate the other memoization related issues. The list so far is:
#10769
#10426
#5538

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

See each of the linked issues

Logs from the workflow controller

See each of the linked issues

Logs from in your workflow's wait container

See each of the linked issues

[JPZ13]

Not sure if this is a bug or an enhancement, but some community members we've spoken to have mentioned that memoization in its current state is not usable. Creating this issue to aggregate feedback and get a fix out

[terrytangyuan]

Great idea. There's also cache GC functionality that can only be enabled via environment variable. We can consider making it a controller flag once memoization feature is more mature.

[mnarozny]

Would it be ok to add this one here as well?
#11219

[Joibel]

Would it be ok to add this one here as well? #11219

Yeah, good to have as many things here as we can whilst we look at it.

[kwohlfahrt]

I'd like to add #10012 as well - users want to be able to manually invalidate a cache (for debugging purposes), but this currently requires giving access to resources in the controller namespace.

IMO, cache configmaps should be created in the workflow's namespace by default rather than the controller's, but allowing a namespace to be specified would remove the worst of the pain.

[mikita-zhuryk]

Hi! I have actually encountered the same problem as in #10426 and I'm not really sure that it's an orchestration templates-related issue as you say here #10769 (comment) (at least the part with retryStrategy). I'm currently looking into this and I think I've found some clues why this happens. If it's OK with you, I'll try to come up with a fix and report my findings in the original issue.

[JPZ13]

For sure @mikita-zhuryk - we'd love you to dive in