Can't run example argo submit artifact-passing.yaml

[dove-young]

Is this a BUG REPORT or FEATURE REQUEST?:

What happened:

Can run example argo submit --serviceaccount argo artifact-passing.yaml

I hit hostPath problem as in this issue #1272 .
Then I configured ContainerRuntimeExecutor: kubelet and bound service account to cluster role which can do hostmount-anyuid.

Then I got this error

status:
  finishedAt: "2019-09-30T08:59:57Z"
  message: 'invalid spec: templates.whalesay.outputs.artifacts.hello-art: kubelet
    executor does not support outputs from base image layer. must use emptyDir'
  phase: Failed

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Argo version:

argo: v2.2.1
  BuildDate: 2018-10-11T16:26:28Z
  GitCommit: 3b52b26190163d1f72f3aef1a39f9f291378dafb
  GitTreeState: clean
  GitTag: v2.2.1
  GoVersion: go1.10.3
  Compiler: gc
  Platform: linux/amd64

• Kubernetes version :

clientVersion:
  buildDate: "2019-05-08T02:18:32Z"
  compiler: gc
  gitCommit: eb4df6c6fb47f5b4fd1ed8bfbfe2d0ed5ea636e1
  gitTreeState: clean
  gitVersion: v1.13.5+icp-ee
  goVersion: go1.11.5
  major: "1"
  minor: "13"
  platform: linux/amd64
serverVersion:
  buildDate: "2019-05-08T02:18:32Z"
  compiler: gc
  gitCommit: eb4df6c6fb47f5b4fd1ed8bfbfe2d0ed5ea636e1
  gitTreeState: clean
  gitVersion: v1.13.5+icp-ee
  goVersion: go1.11.5
  major: "1"
  minor: "13"
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

$ argo get artifact-passing-m782z
Name:                artifact-passing-m782z
Namespace:           argo
ServiceAccount:      argo
Status:              Failed
Message:             invalid spec: templates.whalesay.outputs.artifacts.hello-art: kubelet executor does not support outputs from base image layer. must use emptyDir
Created:             Mon Sep 30 02:09:30 -0700 (30 seconds ago)
Started:             Mon Sep 30 02:09:30 -0700 (30 seconds ago)
Finished:            Mon Sep 30 02:09:30 -0700 (30 seconds ago)
Duration:            0 seconds

• executor logs:

$ kubectl logs <failedpodname> -c init
$ kubectl logs <failedpodname> -c wait

• workflow-controller logs:

$ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)
time="2019-09-30T08:58:46Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=artifact-passing-tjkhn
time="2019-09-30T08:58:46Z" level=info msg="Updated message  -> child 'artifact-passing-tjkhn-2569708171' failed" namespace=argo workflow=artifact-passing-tjkhn
time="2019-09-30T08:58:46Z" level=info msg="Marking workflow completed" namespace=argo workflow=artifact-passing-tjkhn
time="2019-09-30T08:58:46Z" level=info msg="Checking daemoned children of " namespace=argo workflow=artifact-passing-tjkhn
time="2019-09-30T08:58:46Z" level=info msg="Workflow update successful" namespace=argo workflow=artifact-passing-tjkhn
time="2019-09-30T08:59:53Z" level=info msg="Detected ConfigMap update. Updating the controller config."
time="2019-09-30T08:59:53Z" level=info msg="workflow controller configuration from workflow-controller-configmap:\nContainerRuntimeExecutor: kubelet\nartifactRepository:\n  s3:\n    bucket: my-bucket\n    endpoint: argo-artifacts.default:9000\n    insecure: true\n    # accessKeySecret and secretKeySecret are secret selectors.\n    # It references the k8s secret named 'argo-artifacts'\n    # which was created during the minio helm install. The keys,\n    # 'accesskey' and 'secretkey', inside that secret are where the\n    # actual minio credentials are stored.\n    accessKeySecret:\n      name: argo-artifacts\n      key: AKIAIOSFODNN7EXAMPLE\n    secretKeySecret:\n      name: argo-artifacts\n      key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n"
time="2019-09-30T08:59:57Z" level=info msg="Processing workflow" namespace=argo workflow=artifact-passing-sg6tb
time="2019-09-30T08:59:57Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=artifact-passing-sg6tb
time="2019-09-30T08:59:57Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=artifact-passing-sg6tb
time="2019-09-30T08:59:57Z" level=info msg="Updated message  -> invalid spec: templates.whalesay.outputs.artifacts.hello-art: kubelet executor does not support outputs from base image layer. must use emptyDir" namespace=argo workflow=artifact-passing-sg6tb
time="2019-09-30T08:59:57Z" level=info msg="Marking workflow completed" namespace=argo workflow=artifact-passing-sg6tb
time="2019-09-30T08:59:57Z" level=info msg="Checking daemoned children of " namespace=argo workflow=artifact-passing-sg6tb
time="2019-09-30T08:59:57Z" level=info msg="Workflow update successful" namespace=argo workflow=artifact-passing-sg6tb
time="2019-09-30T09:00:04Z" level=info msg="Alloc=4635 TotalAlloc=20147 Sys=71358 NumGC=24 Goroutines=61"
W0930 09:05:03.463213       1 reflector.go:270] github.com/argoproj/argo/workflow/controller/config.go:212: watch of *v1.ConfigMap ended with: too old resource version: 226379 (226449)
time="2019-09-30T09:05:04Z" level=info msg="Alloc=4065 TotalAlloc=20299 Sys=71358 NumGC=27 Goroutines=59"
time="2019-09-30T09:09:30Z" level=info msg="Processing workflow" namespace=argo workflow=artifact-passing-m782z
time="2019-09-30T09:09:30Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=artifact-passing-m782z
time="2019-09-30T09:09:30Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=artifact-passing-m782z
time="2019-09-30T09:09:30Z" level=info msg="Updated message  -> invalid spec: templates.whalesay.outputs.artifacts.hello-art: kubelet executor does not support outputs from base image layer. must use emptyDir" namespace=argo workflow=artifact-passing-m782z
time="2019-09-30T09:09:30Z" level=info msg="Marking workflow completed" namespace=argo workflow=artifact-passing-m782z
time="2019-09-30T09:09:30Z" level=info msg="Checking daemoned children of " namespace=argo workflow=artifact-passing-m782z
time="2019-09-30T09:09:30Z" level=info msg="Workflow update successful" namespace=argo workflow=artifact-passing-m782z
time="2019-09-30T09:10:04Z" level=info msg="Alloc=4515 TotalAlloc=20851 Sys=71358 NumGC=29 Goroutines=61"


W0930 09:10:29.497705       1 reflector.go:270] github.com/argoproj/argo/workflow/controller/config.go:212: watch of *v1.ConfigMap ended with: too old resource version: 227376 (227454)

[logicfox]

May be this will help #1256 (comment)

@jessesuen @sarabala1979 and other leads, I suppose we need more documentation around this. I couldn't find a lot of information about the executor types in the docs and ended up fishing for that comment in the issues.