hostPath type volume gets mounted as emptyDir

[Amey-D]

While trying to deploy a custom model on Seldon Core on minikube, I noticed that if I add volume mount of type hostPath, the resulting pod gets configured with type emptyDir. Is this a known issue?

I am able to repro to issue using the sklearn_iris_deployment.json example, as follows:

$ minikube version
minikube version: v0.25.0
$ minikube ssh

### Inside minikube ssh session:
$ sudo mkdir /data
$ sudo touch /data/foo
$ ls -l /data
total 0
-rw-r--r-- 1 root root 0 Mar 14 21:21 foo

Inside seldon-core/examples/models/sklearn_iris directory:

$ git diff 
diff --git a/examples/models/sklearn_iris/sklearn_iris_deployment.json b/examples/models/sklearn_iris/sklearn_iris_deployment.json
index 08d930e..a7e94ab 100644
--- a/examples/models/sklearn_iris/sklearn_iris_deployment.json
+++ b/examples/models/sklearn_iris/sklearn_iris_deployment.json
@@ -28,9 +28,24 @@
                                     "requests": {
                                         "memory": "1Mi"
                                     }
-                                }
+                                },
+                               "volumeMounts": [
+                                   {
+                                       "mountPath": "/data",
+                                       "name": "data"
+                                   }
+                               ]
                             }
                         ],
+                       "volumes": [
+                           {
+                               "name": "data",
+                               "hostPath": {
+                                   "path": "/data",
+                                   "type": "Directory"
+                               }
+                           }
+                       ],
                         "terminationGracePeriodSeconds": 20
                     }
                 },

$ kubectl apply -f sklearn_iris_deployment.json
seldondeployment "seldon-deployment-example" configured

$ kubectl describe pods sklearn-iris-deployment-sklearn-iris-predictor-7d8ff978b6-85g8p
Name:           sklearn-iris-deployment-sklearn-iris-predictor-7d8ff978b6-85g8p
Namespace:      default
Node:           minikube/192.168.99.100
Start Time:     Wed, 14 Mar 2018 14:43:32 -0700
Labels:         pod-template-hash=3849953462
                seldon-app=sklearn-iris-deployment
Annotations:    prometheus.io/path=/prometheus
                prometheus.io/port=8000
                prometheus.io/scrape=true
Status:         Running
IP:             172.17.0.9
Controlled By:  ReplicaSet/sklearn-iris-deployment-sklearn-iris-predictor-7d8ff978b6
Containers:
  sklearn-iris-classifier:
    Container ID:   docker://9994c1b1283c0012481c171ec650f86edfea7ad227f862352e2ce9cc4f2f0283
    Image:          seldonio/irisclassifier:0.1
    Image ID:       docker://sha256:d327c4e7c32134f2d7668ad97b4e65f61f3e143a01d48c6fb1d374505a871168
    Port:           9000/TCP
    State:          Running
      Started:      Wed, 14 Mar 2018 14:43:33 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      memory:   1Mi
    Liveness:   tcp-socket :http delay=10s timeout=1s period=5s #success=1 #failure=3
    Readiness:  tcp-socket :http delay=10s timeout=1s period=5s #success=1 #failure=3
    Environment:
      PREDICTIVE_UNIT_SERVICE_PORT:  9000
      PREDICTIVE_UNIT_PARAMETERS:    []
      PREDICTIVE_UNIT_ID:            sklearn-iris-classifier
      PREDICTOR_ID:                  sklearn-iris-predictor
      SELDON_DEPLOYMENT_ID:          seldon-deployment-example
    Mounts:
      /data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8plwx (ro)
  seldon-container-engine:
    Container ID:   docker://62bbe0665e8faac51552989dba171e466132676410d9e7d1c5616ef0895678a2
    Image:          seldonio/engine:0.1.5
    Image ID:       docker-pullable://seldonio/engine@sha256:389de5dad7e63774cf90bd46799f425ec7da0dd4f61c997ef10549c40d7463ef
    Ports:          8000/TCP, 8082/TCP
    State:          Running
      Started:      Wed, 14 Mar 2018 14:43:33 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      100m
    Liveness:   http-get http://:admin/ready delay=10s timeout=2s period=10s #success=1 #failure=3
    Readiness:  http-get http://:admin/ready delay=10s timeout=2s period=10s #success=1 #failure=3
    Environment:
      ENGINE_PREDICTOR:          eyJuYW1lIjoic2tsZWFybi1pcmlzLXByZWRpY3RvciIsImdyYXBoIjp7Im5hbWUiOiJza2xlYXJuLWlyaXMtY2xhc3NpZmllciIsInR5cGUiOiJNT0RFTCIsImVuZHBvaW50Ijp7InNlcnZpY2VfaG9zdCI6IjAuMC4wLjAiLCJzZXJ2aWNlX3BvcnQiOjkwMDAsInR5cGUiOiJSRVNUIn19LCJjb21wb25lbnRTcGVjIjp7Im1ldGFkYXRhIjp7ImxhYmVscyI6eyJzZWxkb24tYXBwIjoic2tsZWFybi1pcmlzLWRlcGxveW1lbnQifX0sInNwZWMiOnsidm9sdW1lcyI6W3sibmFtZSI6ImRhdGEifV0sImNvbnRhaW5lcnMiOlt7Im5hbWUiOiJza2xlYXJuLWlyaXMtY2xhc3NpZmllciIsImltYWdlIjoic2VsZG9uaW8vaXJpc2NsYXNzaWZpZXI6MC4xIiwicG9ydHMiOlt7Im5hbWUiOiJodHRwIiwiY29udGFpbmVyUG9ydCI6OTAwMH1dLCJlbnYiOlt7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI5MDAwIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX1BBUkFNRVRFUlMiLCJ2YWx1ZSI6IltdIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lEIiwidmFsdWUiOiJza2xlYXJuLWlyaXMtY2xhc3NpZmllciJ9LHsibmFtZSI6IlBSRURJQ1RPUl9JRCIsInZhbHVlIjoic2tsZWFybi1pcmlzLXByZWRpY3RvciJ9LHsibmFtZSI6IlNFTERPTl9ERVBMT1lNRU5UX0lEIiwidmFsdWUiOiJzZWxkb24tZGVwbG95bWVudC1leGFtcGxlIn1dLCJyZXNvdXJjZXMiOnsicmVxdWVzdHMiOnsibWVtb3J5IjoiMU1pIn19LCJ2b2x1bWVNb3VudHMiOlt7Im5hbWUiOiJkYXRhIiwibW91bnRQYXRoIjoiL2RhdGEifV0sImxpdmVuZXNzUHJvYmUiOnsiaGFuZGxlciI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifX0sImluaXRpYWxEZWxheVNlY29uZHMiOjEwLCJwZXJpb2RTZWNvbmRzIjo1fSwicmVhZGluZXNzUHJvYmUiOnsiaGFuZGxlciI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifX0sImluaXRpYWxEZWxheVNlY29uZHMiOjEwLCJwZXJpb2RTZWNvbmRzIjo1fSwibGlmZWN5Y2xlIjp7InByZVN0b3AiOnsiZXhlYyI6eyJjb21tYW5kIjpbIi9iaW4vc2giLCItYyIsIi9iaW4vc2xlZXAgNSJdfX19LCJpbWFnZVB1bGxQb2xpY3kiOiJJZk5vdFByZXNlbnQifV0sInRlcm1pbmF0aW9uR3JhY2VQZXJpb2RTZWNvbmRzIjoyMH19LCJyZXBsaWNhcyI6MSwiYW5ub3RhdGlvbnMiOnsicHJlZGljdG9yX3ZlcnNpb24iOiIwLjEifX0=
      ENGINE_SELDON_DEPLOYMENT:  eyJhcGlWZXJzaW9uIjoibWFjaGluZWxlYXJuaW5nLnNlbGRvbi5pby92MWFscGhhMSIsImtpbmQiOiJTZWxkb25EZXBsb3ltZW50IiwibWV0YWRhdGEiOnsibmFtZSI6InNlbGRvbi1kZXBsb3ltZW50LWV4YW1wbGUiLCJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VsZkxpbmsiOiIvYXBpcy9tYWNoaW5lbGVhcm5pbmcuc2VsZG9uLmlvL3YxYWxwaGExL25hbWVzcGFjZXMvZGVmYXVsdC9zZWxkb25kZXBsb3ltZW50cy9zZWxkb24tZGVwbG95bWVudC1leGFtcGxlIiwidWlkIjoiOTAxNGI4MzAtMjdjZS0xMWU4LTg5MjMtMDgwMDI3YjY0OTY4IiwicmVzb3VyY2VWZXJzaW9uIjoiMTA1MTUiLCJnZW5lcmF0aW9uIjowLCJjcmVhdGlvblRpbWVzdGFtcCI6IjIwMTgtMDMtMTRUMjE6Mjc6NTZaIiwibGFiZWxzIjp7ImFwcCI6InNlbGRvbiJ9LCJhbm5vdGF0aW9ucyI6eyJrdWJlY3RsLmt1YmVybmV0ZXMuaW8vbGFzdC1hcHBsaWVkLWNvbmZpZ3VyYXRpb24iOiJ7XCJhcGlWZXJzaW9uXCI6XCJtYWNoaW5lbGVhcm5pbmcuc2VsZG9uLmlvL3YxYWxwaGExXCIsXCJraW5kXCI6XCJTZWxkb25EZXBsb3ltZW50XCIsXCJtZXRhZGF0YVwiOntcImFubm90YXRpb25zXCI6e30sXCJsYWJlbHNcIjp7XCJhcHBcIjpcInNlbGRvblwifSxcIm5hbWVcIjpcInNlbGRvbi1kZXBsb3ltZW50LWV4YW1wbGVcIixcIm5hbWVzcGFjZVwiOlwiZGVmYXVsdFwifSxcInNwZWNcIjp7XCJhbm5vdGF0aW9uc1wiOntcImRlcGxveW1lbnRfdmVyc2lvblwiOlwiMC4xXCIsXCJwcm9qZWN0X25hbWVcIjpcIklyaXMgY2xhc3NpZmljYXRpb25cIn0sXCJuYW1lXCI6XCJza2xlYXJuLWlyaXMtZGVwbG95bWVudFwiLFwib2F1dGhfa2V5XCI6XCJvYXV0aC1rZXlcIixcIm9hdXRoX3NlY3JldFwiOlwib2F1dGgtc2VjcmV0XCIsXCJwcmVkaWN0b3JzXCI6W3tcImFubm90YXRpb25zXCI6e1wicHJlZGljdG9yX3ZlcnNpb25cIjpcIjAuMVwifSxcImNvbXBvbmVudFNwZWNcIjp7XCJzcGVjXCI6e1wiY29udGFpbmVyc1wiOlt7XCJpbWFnZVwiOlwic2VsZG9uaW8vaXJpc2NsYXNzaWZpZXI6MC4xXCIsXCJpbWFnZVB1bGxQb2xpY3lcIjpcIklmTm90UHJlc2VudFwiLFwibmFtZVwiOlwic2tsZWFybi1pcmlzLWNsYXNzaWZpZXJcIixcInJlc291cmNlc1wiOntcInJlcXVlc3RzXCI6e1wibWVtb3J5XCI6XCIxTWlcIn19LFwidm9sdW1lTW91bnRzXCI6W3tcIm1vdW50UGF0aFwiOlwiL2RhdGFcIixcIm5hbWVcIjpcImRhdGFcIn1dfV0sXCJ0ZXJtaW5hdGlvbkdyYWNlUGVyaW9kU2Vjb25kc1wiOjIwLFwidm9sdW1lc1wiOlt7XCJob3N0UGF0aFwiOntcInBhdGhcIjpcIi9kYXRhXCIsXCJ0eXBlXCI6XCJEaXJlY3RvcnlcIn0sXCJuYW1lXCI6XCJkYXRhXCJ9XX19LFwiZ3JhcGhcIjp7XCJjaGlsZHJlblwiOltdLFwiZW5kcG9pbnRcIjp7XCJ0eXBlXCI6XCJSRVNUXCJ9LFwibmFtZVwiOlwic2tsZWFybi1pcmlzLWNsYXNzaWZpZXJcIixcInR5cGVcIjpcIk1PREVMXCJ9LFwibmFtZVwiOlwic2tsZWFybi1pcmlzLXByZWRpY3RvclwiLFwicmVwbGljYXNcIjoxfV19fVxuIn0sImNsdXN0ZXJOYW1lIjoiIn0sInNwZWMiOnsibmFtZSI6InNrbGVhcm4taXJpcy1kZXBsb3ltZW50IiwicHJlZGljdG9ycyI6W3sibmFtZSI6InNrbGVhcm4taXJpcy1wcmVkaWN0b3IiLCJncmFwaCI6eyJuYW1lIjoic2tsZWFybi1pcmlzLWNsYXNzaWZpZXIiLCJ0eXBlIjoiTU9ERUwiLCJlbmRwb2ludCI6eyJzZXJ2aWNlX2hvc3QiOiIwLjAuMC4wIiwic2VydmljZV9wb3J0Ijo5MDAwLCJ0eXBlIjoiUkVTVCJ9fSwiY29tcG9uZW50U3BlYyI6eyJtZXRhZGF0YSI6eyJsYWJlbHMiOnsic2VsZG9uLWFwcCI6InNrbGVhcm4taXJpcy1kZXBsb3ltZW50In19LCJzcGVjIjp7InZvbHVtZXMiOlt7Im5hbWUiOiJkYXRhIn1dLCJjb250YWluZXJzIjpbeyJuYW1lIjoic2tsZWFybi1pcmlzLWNsYXNzaWZpZXIiLCJpbWFnZSI6InNlbGRvbmlvL2lyaXNjbGFzc2lmaWVyOjAuMSIsInBvcnRzIjpbeyJuYW1lIjoiaHR0cCIsImNvbnRhaW5lclBvcnQiOjkwMDB9XSwiZW52IjpbeyJuYW1lIjoiUFJFRElDVElWRV9VTklUX1NFUlZJQ0VfUE9SVCIsInZhbHVlIjoiOTAwMCJ9LHsibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9QQVJBTUVURVJTIiwidmFsdWUiOiJbXSJ9LHsibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9JRCIsInZhbHVlIjoic2tsZWFybi1pcmlzLWNsYXNzaWZpZXIifSx7Im5hbWUiOiJQUkVESUNUT1JfSUQiLCJ2YWx1ZSI6InNrbGVhcm4taXJpcy1wcmVkaWN0b3IifSx7Im5hbWUiOiJTRUxET05fREVQTE9ZTUVOVF9JRCIsInZhbHVlIjoic2VsZG9uLWRlcGxveW1lbnQtZXhhbXBsZSJ9XSwicmVzb3VyY2VzIjp7InJlcXVlc3RzIjp7Im1lbW9yeSI6IjFNaSJ9fSwidm9sdW1lTW91bnRzIjpbeyJuYW1lIjoiZGF0YSIsIm1vdW50UGF0aCI6Ii9kYXRhIn1dLCJsaXZlbmVzc1Byb2JlIjp7ImhhbmRsZXIiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn19LCJpbml0aWFsRGVsYXlTZWNvbmRzIjoxMCwicGVyaW9kU2Vjb25kcyI6NX0sInJlYWRpbmVzc1Byb2JlIjp7ImhhbmRsZXIiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn19LCJpbml0aWFsRGVsYXlTZWNvbmRzIjoxMCwicGVyaW9kU2Vjb25kcyI6NX0sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDUiXX19fSwiaW1hZ2VQdWxsUG9saWN5IjoiSWZOb3RQcmVzZW50In1dLCJ0ZXJtaW5hdGlvbkdyYWNlUGVyaW9kU2Vjb25kcyI6MjB9fSwicmVwbGljYXMiOjEsImFubm90YXRpb25zIjp7InByZWRpY3Rvcl92ZXJzaW9uIjoiMC4xIn19XSwib2F1dGhfa2V5Ijoib2F1dGgta2V5Iiwib2F1dGhfc2VjcmV0Ijoib2F1dGgtc2VjcmV0IiwiYW5ub3RhdGlvbnMiOnsiZGVwbG95bWVudF92ZXJzaW9uIjoiMC4xIiwicHJvamVjdF9uYW1lIjoiSXJpcyBjbGFzc2lmaWNhdGlvbiJ9fX0=
      ENGINE_SERVER_PORT:        8000
      ENGINE_SERVER_GRPC_PORT:   5001
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8plwx (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  data:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-8plwx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8plwx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                 Age   From               Message
  ----     ------                 ----  ----               -------
  Normal   Scheduled              54s   default-scheduler  Successfully assigned sklearn-iris-deployment-sklearn-iris-predictor-7d8ff978b6-85g8p to minikube
  Normal   SuccessfulMountVolume  54s   kubelet, minikube  MountVolume.SetUp succeeded for volume "data"
  Normal   SuccessfulMountVolume  54s   kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-8plwx"
  Normal   Pulled                 53s   kubelet, minikube  Container image "seldonio/irisclassifier:0.1" already present on machine
  Normal   Created                53s   kubelet, minikube  Created container
  Normal   Started                53s   kubelet, minikube  Started container
  Normal   Pulled                 53s   kubelet, minikube  Container image "seldonio/engine:0.1.5" already present on machine
  Normal   Created                53s   kubelet, minikube  Created container
  Normal   Started                53s   kubelet, minikube  Started container
  Warning  Unhealthy              42s   kubelet, minikube  Liveness probe failed: Get http://172.17.0.9:8082/ready: dial tcp 172.17.0.9:8082: getsockopt: connection refused 

The deployment is functional otherwise i.e. I can run curl commands on [1] and retrieve predictions.

References:
[1] https://github.com/SeldonIO/seldon-core/blob/master/docs/getting_started/minikube.md
[2] https://kubernetes.io/docs/concepts/storage/volumes/#hostpath

[ukclivecox]

At present you need to use the form with "volumeSource"

"volumes": [
			    {
				"name": "data",
				"volumeSource": {
				    "hostPath": {
					"path": "/data",
					"type": "Directory"
				    }
				}
			    }
			],

This is due to the fact we use the k8s proto buffers to parse the PodTemplateSpec and it needs the explicit form of declaring volumes using volumeSource.