Memoization doesn't work in templates that are dag or use retries #10426

gmeshkov · 2023-01-30T13:46:43Z

Pre-requisites

I have double-checked my configuration
I can confirm the issues exists when I tested with :latest
I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

When I add memoization to a step with retryStrategy, the memoization doesn't work.
Checked with retryStrategy: Always and the default OnFailure value.

Also, the memoization doesn't work for a step which is a dag containing other steps.

From the logs of the controller I can conclude that the entry in ConfigMap is not created for some reason after the retry/dag steps finish.

Version

3.4.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

# This test shows that memoization doesn't work for templates with retryStrategy and dag templates
# `regular-1` and `regular-2` tasks show that memoization works for regular templates
# `with-retries-1` and `with-retries-2` show that if you add the retryStrategy to the template, memoization no longer works
# `with-dag-1` and `with-dag-2` show that memoization doesn't work if a memoized template contains multiple steps (is a dag)
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: memoized-workflow-test
spec:
  entrypoint: main
#  podGC:
#    strategy: OnPodCompletion
  templates:
  - name: main
    dag:
      tasks:
      - name: regular-1
        template: run
        arguments:
          parameters:
          - name: id
            value: 1
          - name: cache-key
            value: '{{workflow.name}}'
      - name: regular-2
        template: run
        depends: regular-1.Succeeded
        arguments:
          parameters:
          - name: id
            value: 2
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-retries-1
        template: run-with-retries
        arguments:
          parameters:
          - name: id
            value: 3
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-retries-2
        template: run-with-retries
        depends: with-retries-1.Succeeded
        arguments:
          parameters:
          - name: id
            value: 4
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-dag-1
        template: run-with-dag
        arguments:
          parameters:
          - name: id
            value: 5
          - name: cache-key
            value: '{{workflow.name}}'
      - name: with-dag-2
        template: run-with-dag
        depends: with-dag-1.Succeeded
        arguments:
          parameters:
          - name: id
            value: 6
          - name: cache-key
            value: '{{workflow.name}}'

  - name: run
    inputs:
      parameters:
      - name: id
      - name: cache-key
    script:
      image: ubuntu:22.04
      command: [bash]
      source: |
        sleep 30
        echo result: {{inputs.parameters.id}}
    memoize:
      key: "regular-{{inputs.parameters.cache-key}}"
      cache:
        configMap:
          name: memoization-test-cache

  - name: run-with-retries
    inputs:
      parameters:
      - name: id
      - name: cache-key
    script:
      image: ubuntu:22.04
      command: [bash]
      source: |
        sleep 30
        echo result: {{inputs.parameters.id}}
    memoize:
      key: "retry-{{inputs.parameters.cache-key}}"
      cache:
        configMap:
          name: memoization-test-cache
    retryStrategy:
      limit: '1'
      retryPolicy: Always

  - name: run-raw
    inputs:
      parameters:
      - name: id
      - name: cache-key
    script:
      image: ubuntu:22.04
      command: [bash]
      source: |
        sleep 30
        echo result: {{inputs.parameters.id}}

  - name: run-with-dag
    inputs:
      parameters:
      - name: id
      - name: cache-key
    dag:
      tasks:
      - name: run-raw-step
        template: run-raw
        arguments:
          parameters:
          - name: id
            value: '{{inputs.parameters.id}}'
          - name: cache-key
            value: '{{inputs.parameters.cache-key}}'
    memoize:
      key: "dag-{{inputs.parameters.cache-key}}"
      cache:
        configMap:
          name: memoization-test-cache

Logs from the workflow controller

time="2023-01-30T13:36:36.431Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.461Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.461Z" level=info msg="DAG node memoized-workflow-testwttf6 initialized Running" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.461Z" level=info msg="All of node memoized-workflow-testwttf6.regular-1 dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.565Z" level=info msg="Pod node memoized-workflow-testwttf6-2651566501 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.624Z" level=info msg="Created pod: memoized-workflow-testwttf6.regular-1 (memoized-workflow-testwttf6-run-2651566501)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.624Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-1 dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.661Z" level=info msg="DAG node memoized-workflow-testwttf6-3689047750 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.661Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-1.run-raw-step dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.661Z" level=info msg="Pod node memoized-workflow-testwttf6-1860721307 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.773Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-dag-1.run-raw-step (memoized-workflow-testwttf6-run-raw-1860721307)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.773Z" level=info msg="All of node memoized-workflow-testwttf6.with-retries-1 dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.962Z" level=info msg="Retry node memoized-workflow-testwttf6-412563200 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:36.962Z" level=info msg="Pod node memoized-workflow-testwttf6-643257443 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.126Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-retries-1(0) (memoized-workflow-testwttf6-run-with-retries-643257443)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.126Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.126Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:37.137Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314195848 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.625Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-643257443 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-2651566501 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.626Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-1860721307 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.627Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.627Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:36:46.638Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314197033 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.984Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.985Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-643257443 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.986Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.986Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.986Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.987Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:23.991Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/memoized-workflow-testwttf6-run-with-retries-643257443/terminateContainers
time="2023-01-30T13:37:23.998Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314200811 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.993Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.994Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-643257443 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node memoized-workflow-testwttf6-412563200 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="node memoized-workflow-testwttf6-412563200 finished: 2023-01-30 13:37:33.99591818 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:33.995Z" level=info msg="All of node memoized-workflow-testwttf6.with-retries-2 dependencies [with-retries-1] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.002Z" level=info msg="Retry node memoized-workflow-testwttf6-462896057 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.002Z" level=info msg="Pod node memoized-workflow-testwttf6-2914119264 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.019Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-retries-2(0) (memoized-workflow-testwttf6-run-with-retries-2914119264)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.019Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.019Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:34.031Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314201878 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.020Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-2914119264 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.021Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.022Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.022Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:44.034Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314202945 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:53.992Z" level=info msg="cleaning up pod" action=killContainers key=argo/memoized-workflow-testwttf6-run-with-retries-643257443/killContainers
time="2023-01-30T13:37:57.960Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-1860721307 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.961Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-2651566501 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="Outbound nodes of memoized-workflow-testwttf6-3689047750 set to [memoized-workflow-testwttf6-1860721307]" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="node memoized-workflow-testwttf6-3689047750 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="node memoized-workflow-testwttf6-3689047750 finished: 2023-01-30 13:37:57.962392225 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="Checking daemoned children of memoized-workflow-testwttf6-3689047750" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.962Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-2 dependencies [with-dag-1] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.967Z" level=info msg="cleaning up pod" action=terminateContainers key=argo/memoized-workflow-testwttf6-run-2651566501/terminateContainers
time="2023-01-30T13:37:57.967Z" level=info msg="DAG node memoized-workflow-testwttf6-3672270131 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.968Z" level=info msg="All of node memoized-workflow-testwttf6.with-dag-2.run-raw-step dependencies [] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.968Z" level=info msg="Pod node memoized-workflow-testwttf6-3956060252 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.987Z" level=info msg="Created pod: memoized-workflow-testwttf6.with-dag-2.run-raw-step (memoized-workflow-testwttf6-run-raw-3956060252)" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.988Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:57.988Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:37:58.000Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314204100 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.990Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-2651566501 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:07.992Z" level=info msg="Saving ConfigMap cache entry" key=regular-memoized-workflow-testwttf6 name=memoization-test-cache namespace=argo nodeId=memoized-workflow-testwttf6-2651566501
time="2023-01-30T13:38:08.003Z" level=info msg="node changed" namespace=argo new.message= new.phase=Running new.progress=0/1 nodeID=memoized-workflow-testwttf6-3956060252 old.message= old.phase=Pending old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.003Z" level=info msg="All of node memoized-workflow-testwttf6.regular-2 dependencies [regular-1] completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.015Z" level=info msg="Pod node memoized-workflow-testwttf6-2601233644 initialized Pending" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.016Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.016Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:08.029Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314205057 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.067Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.069Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=4 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.069Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.069Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-2914119264 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-3956060252 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.070Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg="node memoized-workflow-testwttf6-462896057 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg="node memoized-workflow-testwttf6-462896057 finished: 2023-01-30 13:38:20.071530014 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.071Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:20.085Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=314206244 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:27.968Z" level=info msg="cleaning up pod" action=killContainers key=argo/memoized-workflow-testwttf6-run-2651566501/killContainers
time="2023-01-30T13:38:30.087Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=4 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-3956060252 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.089Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.090Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:30.090Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.387Z" level=info msg="Processing workflow" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=5 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="task-result changed" namespace=argo nodeID=memoized-workflow-testwttf6-3956060252 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=memoized-workflow-testwttf6-3956060252 old.message= old.phase=Running old.progress=0/1 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.389Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-1860721307 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.390Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-643257443 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.390Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2651566501 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.390Z" level=info msg="node unchanged" namespace=argo nodeID=memoized-workflow-testwttf6-2914119264 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Outbound nodes of memoized-workflow-testwttf6-3672270131 set to [memoized-workflow-testwttf6-3956060252]" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6-3672270131 phase Pending -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6-3672270131 finished: 2023-01-30 13:38:42.391258267 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Checking daemoned children of memoized-workflow-testwttf6-3672270131" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Outbound nodes of memoized-workflow-testwttf6 set to [memoized-workflow-testwttf6-2601233644 memoized-workflow-testwttf6-3956060252 memoized-workflow-testwttf6-2914119264]" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6 phase Running -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="node memoized-workflow-testwttf6 finished: 2023-01-30 13:38:42.391603635 +0000 UTC" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Checking daemoned children of memoized-workflow-testwttf6" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg=reconcileAgentPod namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Updated phase Running -> Succeeded" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Marking workflow completed" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Marking workflow as pending archiving" namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.391Z" level=info msg="Checking daemoned children of " namespace=argo workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.397Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-1340600742-agent/deletePod
time="2023-01-30T13:38:42.405Z" level=info msg="Workflow update successful" namespace=argo phase=Succeeded resourceVersion=314208768 workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.643Z" level=info msg="archiving workflow" namespace=argo uid=3b66145e-a4ba-4045-adec-c9ada69395fa workflow=memoized-workflow-testwttf6
time="2023-01-30T13:38:42.669Z" level=info msg="Queueing Succeeded workflow argo/memoized-workflow-testwttf6 for delete in 2h0m0s due to TTL"
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-2651566501/deletePod
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-raw-1860721307/deletePod
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-with-retries-2914119264/deletePod
time="2023-01-30T13:38:47.643Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-with-retries-643257443/deletePod
time="2023-01-30T13:38:47.657Z" level=info msg="cleaning up pod" action=deletePod key=argo/memoized-workflow-testwttf6-run-raw-3956060252/deletePod

Logs from in your workflow's wait container

No resources found in argo namespace.

The text was updated successfully, but these errors were encountered:

stale · 2023-03-25T11:08:08Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

Despite argoproj#10769 and argoproj#10426 both having examples of memoization not working with the examples having no output, no-one has picked up on this. To address this improve the documentation for memoization and work-avoidance, linking the two ideas and pointing people who want to skip steps towards work-avoidance unless they are really doing what memoization was designed to do. Issue argoproj#10426 is problematic in that some steps get memoized when perhaps they should't, so this commit shouldn't close it. Fixes argoproj#10769 Signed-off-by: Alan Clucas <[email protected]>

Please note: I am not totally in favour of this change, it makes memoization work without outputs and I'm not conviced this is actually healthy, but it's what people seem to expect. See the linked issues in the fixes list below. This change will cause memoization to work for all step and dag tasks even without outputs. Note: They were working semi-erroneously for some dag tasks due to implicit outputs before this change. Fixes argoproj#11280 (raised to cover this desire) Fixes argoproj#10769 (already closed by documentation) Partially addresses argoproj#10426: Dags will memoize now, but retries still won't Signed-off-by: Alan Clucas <[email protected]>

Joibel · 2023-07-25T14:09:03Z

This should be improved but not fixed in 3.5, via #11379. The dag case now memoizes, but the retries don't.

stale · 2023-09-17T03:52:44Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

: #12490 #10426 (#12491)

: argoproj#12490 argoproj#10426 (argoproj#12491)

: argoproj#12490 argoproj#10426 (argoproj#12491) Signed-off-by: Isitha Subasinghe <[email protected]>

gmeshkov added the type/bug label Jan 30, 2023

sarabala1979 added the P3 Low priority label Jan 30, 2023

caelan-io added the area/memoization label Feb 23, 2023

stale bot added the problem/stale This has not had a response in some time label Mar 25, 2023

moleskin-smile mentioned this issue Mar 28, 2023

Memoization doesn't work in templates that are steps #10769

Closed

3 tasks

JPZ13 added P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important and removed P3 Low priority labels Mar 30, 2023

stale bot removed problem/stale This has not had a response in some time labels Mar 30, 2023

JPZ13 mentioned this issue Jun 8, 2023

Memoization isn't feature complete #11191

Open

3 tasks

Joibel mentioned this issue Jun 23, 2023

docs: Link memoization and work-avoidance #11257

Merged

juliev0 added P1 High priority. All bugs with >=5 thumbs up that aren’t P0, plus: Any other bugs deemed high priority and removed P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important labels Jun 29, 2023

Joibel mentioned this issue Jul 17, 2023

feat: Allow memoization without outputs #11379

Merged

stale bot added the problem/stale This has not had a response in some time label Sep 17, 2023

terrytangyuan removed the problem/stale This has not had a response in some time label Sep 20, 2023

agilgur5 added area/templates/dag area/retry-manual Manual workflow "Retry" Action (API/CLI/UI). See retryStrategy for template-level retries labels Nov 2, 2023

This was referenced Jan 9, 2024

Memoized didn't created when there is retryStrategy #12490

Closed

fix: cache configmap don't create with workflow has retrystrategy. Fixes: #12490 #10426 #12491

Merged

Joibel self-assigned this Jan 9, 2024

terrytangyuan pushed a commit that referenced this issue Jan 19, 2024

fix: cache configmap don't create with workflow has retrystrategy. Fixes

46c1324

: #12490 #10426 (#12491)

terrytangyuan closed this as completed in #12491 Jan 19, 2024

isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue Feb 20, 2024

fix: cache configmap don't create with workflow has retrystrategy. Fixes

1afc2fa

: argoproj#12490 argoproj#10426 (argoproj#12491)

isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue Feb 27, 2024

fix: cache configmap don't create with workflow has retrystrategy. Fixes

7e2a34f

: argoproj#12490 argoproj#10426 (argoproj#12491)

isubasinghe mentioned this issue Feb 27, 2024

Release v3.5 patch releases discussion #11997

Open

isubasinghe pushed a commit to isubasinghe/argo-workflows that referenced this issue Feb 28, 2024

fix: cache configmap don't create with workflow has retrystrategy. Fixes

8c75a72

: argoproj#12490 argoproj#10426 (argoproj#12491) Signed-off-by: Isitha Subasinghe <[email protected]>

agilgur5 added area/retryStrategy Template-level retryStrategy and removed area/retry-manual Manual workflow "Retry" Action (API/CLI/UI). See retryStrategy for template-level retries labels Apr 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Memoization doesn't work in templates that are dag or use retries #10426

Memoization doesn't work in templates that are dag or use retries #10426

gmeshkov commented Jan 30, 2023 •

edited

Loading

stale bot commented Mar 25, 2023

Joibel commented Jul 25, 2023

stale bot commented Sep 17, 2023

Memoization doesn't work in templates that are dag or use retries #10426

Memoization doesn't work in templates that are dag or use retries #10426

Comments

gmeshkov commented Jan 30, 2023 • edited Loading

Pre-requisites

What happened/what you expected to happen?

Version

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Logs from the workflow controller

Logs from in your workflow's wait container

stale bot commented Mar 25, 2023

Joibel commented Jul 25, 2023

stale bot commented Sep 17, 2023

gmeshkov commented Jan 30, 2023 •

edited

Loading