[Question] How can I let exit handler do the clean up after we stop a workflow via API?

[cy-zheng]

Summary

How can I let exit handler do the clean up after we stop a workflow via API?
I find that if we stop/terminate a workflow, the exit handler would never run, so we need to clean up some resources manually.

Motivation

We are using argo resource API to create some native kubernetes objects, we hope that when we stop the workflow, these k8s objects would be deleted by exit handler.

[cy-zheng]

E.g.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: exit-handlers-
spec:
  entrypoint: intentional-fail
  onExit: exit-handler                  # invoke exit-hander template at end of the workflow
  templates:
  # primary workflow template
  - name: intentional-fail
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["sleep 300"]

  # Exit handler templates
  # After the completion of the entrypoint template, the status of the
  # workflow is made available in the global variable {{workflow.status}}.
  # {{workflow.status}} will be one of: Succeeded, Failed, Error
  - name: exit-handler
    steps:
    - - name: notify
        template: send-email
      - name: celebrate
        template: celebrate
        when: "{{workflow.status}} == Succeeded"
      - name: cry
        template: cry
        when: "{{workflow.status}} != Succeeded"
  - name: send-email
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo send e-mail: {{workflow.name}} {{workflow.status}}"]
  - name: celebrate
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo hooray!"]
  - name: cry
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["echo boohoo!"]

Send a stop/terminate request to argo server, and you will find exit handlers failed.

Argo version: v2.7.5

[cy-zheng]

#2710

It seems fixed in v2.8.0