Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Endpoint type is missing for AB-test #451

Closed
sasvaritoni opened this issue Feb 19, 2019 · 6 comments
Closed

Endpoint type is missing for AB-test #451

sasvaritoni opened this issue Feb 19, 2019 · 6 comments

Comments

@sasvaritoni
Copy link
Contributor

I deployed a single model, an AB-test and a MAB with the v1alpha2 jsonnet templates of Kubeflow.

Now, I want to list the endpoint types of the deployments (not for the subcomponents, but the "external endpoint" of the whole SeldonDeployment)

For single model and MAB it's there under "graph":
"graph": {
"endpoint": {
"service_host": ...
"service_port": ...
"type": "REST"
},

But for AB-test it is empty:
"graph": {
"endpoint": {},

I would expect that it should be filled in for AB-test deployments as well.
I am not sure if this originates from the .jsonnet templates used or from the CRD handler.

Also the status for all my SeldonDeployments seems to be stuck in "creating", even after the deployments have been working fine for several weeks:
"status": {
"state": "Creating"
}
Is this some known issue?

Thanks!
@ukclivecox
Copy link
Contributor

  • It certainly should not be stuck in "creating". Can you give us the version of Seldon Core you are using. Can you also look for any errors in the cluster-manager. Is the error repeatable.
  • The basic AB test component is an internal component so there is no external endpoint.

@sasvaritoni
Copy link
Contributor Author

sasvaritoni commented Feb 19, 2019 •
edited
Loading

  • It certainly should not be stuck in "creating". Can you give us the version of Seldon Core you are using. Can you also look for any errors in the cluster-manager. Is the error repeatable.

Seldon-core version is 0.2.3. All deployments I create aremain in "creating" state.

I can see "Unexpected error occurred in scheduled task." messages printed periodically in the logs:

14:55:26.034 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:55:26
14:55:26.034 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 4239485 in namespace mxe
14:55:36.082 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:4239485 Processed:4239485
14:55:36.082 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:55:36
14:55:36.082 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:55:36.144 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:55:41.083 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:55:41
14:55:41.083 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 4239485 in namespace mxe
14:55:51.132 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:4239485 Processed:4239485
14:55:51.132 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:55:51
14:55:51.132 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:55:51.191 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:55:56.133 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:55:56
14:55:56.133 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 4239485 in namespace mxe
14:55:57.427 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - ADDED
 : {"apiVersion":"machinelearning.seldon.io/v1alpha2","kind":"SeldonDeployment","metadata":{"annotations":{"ksonnet.io/managed":"{\"pristine\":\"H4sIAAAAAAAA/8yS0U7dPAzH77/H8HW/Q7sJpPUaNE0a29GKdjMhZBLTeiR2lKQVR6jvPqWFjqPDA+wusePf33/Hz4CBf1JMrAIteDQDCznCKCz9LpGzKjvWs6lBFwb8ABU8slhooVtylxScHjxJhgo8ZbSYEdpncHhPLpUThgAtrKhSnlSEcoEa9UGl1LaQKeUG5goEPW339ZoCmhLzT1RepEBmAYtoxswqi47dWrmbNkdTYYSov8nkuyP0qVSIZNlkjQnaXyf4LXtMnyvYbHSBzFr72qJRychCL0j22BfFyT/9T4TNxaf2vK7rM/b93dT3zUU71bt6V5pZnu5H5/bq2ByghS8P3zTvI6V12I4nEkppH/WeitYDshsj3QyR0qDOQvuxggHFOooln03o1DxSXtxoLFMfcg4wzxWwcGZ0l+Tw0JFRsQnapq4rCBRZ7RY7ryCNxlBKb4SaCjJ70jH/rT2dbyS0/I+3PKkbPV3rKLl82e18++YrOjKR1ngFmaJnWfbjc0RD+2PR5pW1YgqnjxiGZSsGdjaSrCASG5Rl8ZgPoUzsx1V3886CvmSvv19efX0nHSk4Nrg4uZ3n//4AAAD//wEAAP//IkEGlN0DAAA\u003d\"}"},"clusterName":"","creationTimestamp":"2019-02-19T14:55:57Z","generation":1.0,"labels":{"app":"seldon","app.kubernetes.io/deploy-manager":"ksonnet","ksonnet.io/component":"test1"},"name":"test1","namespace":"mxe","resourceVersion":"19472316","selfLink":"/apis/machinelearning.seldon.io/v1alpha2/namespaces/mxe/seldondeployments/test1","uid":"76a9108e-3456-11e9-987f-0050569a614d"},"spec":{"annotations":{"deployment_version":"v1","project_name":"test1"},"name":"test1","predictors":[{"annotations":{"predictor_version":"v1"},"componentSpecs":[{"spec":{"containers":[{"image":"vmx-eea169:5000/img_vgg16:v0.0.1","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":3.0,"handler":{"tcpSocket":{"port":"http"}},"initialDelaySeconds":100.0,"periodSeconds":5.0,"successThreshold":1.0,"timeoutSeconds":1.0},"name":"test1","readinessProbe":{"failureThreshold":3.0,"handler":{"tcpSocket":{"port":"http"}},"initialDelaySeconds":100.0,"periodSeconds":5.0,"successThreshold":1.0,"timeoutSeconds":1.0},"volumeMounts":[]}],"imagePullSecrets":[],"terminationGracePeriodSeconds":1.0,"volumes":[]}}],"graph":{"children":[],"endpoint":{"type":"REST"},"name":"test1","type":"MODEL"},"name":"test1","replicas":1.0}]}}

14:55:57.429 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Running updates for test1
14:55:57.430 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Will try to call LIST /apis/extensions/v1beta1/namespaces/mxe/deployments/test1-test1-svc-orch
14:55:57.491 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - About to CREATE {
  "metadata": {
    "name": "test1-test1-svc-orch",
    "labels": {
      "seldon-app": "test1",
      "seldon-deployment-id": "test1",
      "app": "test1-test1-svc-orch",
      "version": "v1",
      "seldon-type": "deployment"
    },
    "ownerReferences": [{
      "kind": "SeldonDeployment",
      "name": "test1",
      "uid": "76a9108e-3456-11e9-987f-0050569a614d",
      "apiVersion": "machinelearning.seldon.io/v1alpha2",
      "controller": true
    }]
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "seldon-app": "test1"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "seldon-app": "test1",
          "app": "test1-test1-svc-orch"
        },
        "annotations": {
          "deployment_version": "v1",
          "project_name": "test1",
          "predictor_version": "v1",
          "prometheus.io/path": "/prometheus",
          "prometheus.io/port": "8000",
          "prometheus.io/scrape": "true"
        }
      },
      "spec": {
        "volumes": [{
          "name": "podinfo",
          "volumeSource": {
            "downwardAPI": {
              "items": [{
                "path": "annotations",
                "fieldRef": {
                  "fieldPath": "metadata.annotations"
                }
              }]
            }
          }
        }],
        "containers": [{
          "name": "seldon-container-engine",
          "image": "seldonio/engine:0.2.3",
          "ports": [{
            "containerPort": 8000
          }, {
            "name": "admin",
            "containerPort": 8082
          }, {
            "name": "jmx",
            "containerPort": 9090
          }],
          "env": [{
            "name": "ENGINE_PREDICTOR",
            "value": "eyJuYW1lIjoidGVzdDEiLCJncmFwaCI6eyJuYW1lIjoidGVzdDEiLCJ0eXBlIjoiTU9ERUwiLCJlbmRwb2ludCI6eyJzZXJ2aWNlX2hvc3QiOiJ0ZXN0MS10ZXN0MS10ZXN0MSIsInNlcnZpY2VfcG9ydCI6OTAwMCwidHlwZSI6IlJFU1QifX0sImNvbXBvbmVudFNwZWNzIjpbeyJtZXRhZGF0YSI6eyJsYWJlbHMiOnsic2VsZG9uLWFwcC10ZXN0MSI6InRlc3QxLXRlc3QxLXRlc3QxIn19LCJzcGVjIjp7ImNvbnRhaW5lcnMiOlt7Im5hbWUiOiJ0ZXN0MSIsImltYWdlIjoidm14LWVlYTE2OTo1MDAwL2ltZ192Z2cxNjp2MC4wLjEiLCJwb3J0cyI6W3sibmFtZSI6Imh0dHAiLCJjb250YWluZXJQb3J0Ijo5MDAwfV0sImVudiI6W3sibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9TRVJWSUNFX1BPUlQiLCJ2YWx1ZSI6IjkwMDAifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfUEFSQU1FVEVSUyIsInZhbHVlIjoiW10ifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfSUQiLCJ2YWx1ZSI6InRlc3QxIn0seyJuYW1lIjoiUFJFRElDVE9SX0lEIiwidmFsdWUiOiJ0ZXN0MSJ9LHsibmFtZSI6IlNFTERPTl9ERVBMT1lNRU5UX0lEIiwidmFsdWUiOiJ0ZXN0MSJ9XSwidm9sdW1lTW91bnRzIjpbeyJuYW1lIjoicG9kaW5mbyIsInJlYWRPbmx5Ijp0cnVlLCJtb3VudFBhdGgiOiIvZXRjL3BvZGluZm8ifV0sImxpdmVuZXNzUHJvYmUiOnsiaGFuZGxlciI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifX0sImluaXRpYWxEZWxheVNlY29uZHMiOjEwMCwidGltZW91dFNlY29uZHMiOjEsInBlcmlvZFNlY29uZHMiOjUsInN1Y2Nlc3NUaHJlc2hvbGQiOjEsImZhaWx1cmVUaHJlc2hvbGQiOjN9LCJyZWFkaW5lc3NQcm9iZSI6eyJoYW5kbGVyIjp7InRjcFNvY2tldCI6eyJwb3J0IjoiaHR0cCJ9fSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6MTAwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDUiXX19fSwiaW1hZ2VQdWxsUG9saWN5IjoiSWZOb3RQcmVzZW50In1dLCJ0ZXJtaW5hdGlvbkdyYWNlUGVyaW9kU2Vjb25kcyI6MX19XSwicmVwbGljYXMiOjEsImFubm90YXRpb25zIjp7InByZWRpY3Rvcl92ZXJzaW9uIjoidjEifX0="
          }, {
            "name": "DEPLOYMENT_NAME",
            "value": "test1"
          }, {
            "name": "ENGINE_SERVER_PORT",
            "value": "8000"
          }, {
            "name": "ENGINE_SERVER_GRPC_PORT",
            "value": "5001"
          }, {
            "name": "JAVA_OPTS",
            "value": "-Dcom.sun.management.jmxremote.rmi.port=9090 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=127.0.0.1"
          }],
          "resources": {
            "requests": {
              "cpu": "0.1"
            }
          },
          "volumeMounts": [{
            "name": "podinfo",
            "readOnly": true,
            "mountPath": "/etc/podinfo"
          }],
          "livenessProbe": {
            "handler": {
              "httpGet": {
                "path": "/ready",
                "port": "admin"
              }
            },
            "initialDelaySeconds": 10,
            "timeoutSeconds": 2,
            "periodSeconds": 10,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "readinessProbe": {
            "handler": {
              "httpGet": {
                "path": "/ready",
                "port": "admin"
              }
            },
            "initialDelaySeconds": 10,
            "timeoutSeconds": 2,
            "periodSeconds": 10,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "lifecycle": {
            "preStop": {
              "exec": {
                "command": ["/bin/sh", "-c", "curl 127.0.0.1:8000/pause && /bin/sleep 5"]
              }
            }
          },
          "imagePullPolicy": "IfNotPresent",
          "securityContext": {
            "runAsUser": 8888
          }
        }],
        "terminationGracePeriodSeconds": 20
      }
    },
    "strategy": {
      "rollingUpdate": {
        "maxUnavailable": "10%"
      }
    }
  }
}
14:55:57.550 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Created deployment:{
  "metadata": {
    "name": "test1-test1-svc-orch",
    "generateName": "",
    "namespace": "mxe",
    "selfLink": "/apis/extensions/v1beta1/namespaces/mxe/deployments/test1-test1-svc-orch",
    "uid": "76b55bb3-3456-11e9-987f-0050569a614d",
    "resourceVersion": "19472317",
    "generation": 1,
    "creationTimestamp": "2019-02-19T14:55:57Z",
    "labels": {
      "app": "test1-test1-svc-orch",
      "seldon-app": "test1",
      "seldon-deployment-id": "test1",
      "seldon-type": "deployment",
      "version": "v1"
    },
    "ownerReferences": [{
      "kind": "SeldonDeployment",
      "name": "test1",
      "uid": "76a9108e-3456-11e9-987f-0050569a614d",
      "apiVersion": "machinelearning.seldon.io/v1alpha2",
      "controller": true
    }],
    "clusterName": ""
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "seldon-app": "test1"
      }
    },
    "template": {
      "metadata": {
        "name": "",
        "generateName": "",
        "namespace": "",
        "selfLink": "",
        "uid": "",
        "resourceVersion": "",
        "generation": 0,
        "creationTimestamp": "1970-01-01T00:00:00Z",
        "labels": {
          "app": "test1-test1-svc-orch",
          "seldon-app": "test1"
        },
        "annotations": {
          "deployment_version": "v1",
          "predictor_version": "v1",
          "project_name": "test1",
          "prometheus.io/path": "/prometheus",
          "prometheus.io/port": "8000",
          "prometheus.io/scrape": "true"
        },
        "clusterName": ""
      },
      "spec": {
        "volumes": [{
          "name": "podinfo",
          "volumeSource": {
            "downwardAPI": {
              "items": [{
                "path": "annotations",
                "fieldRef": {
                  "apiVersion": "v1",
                  "fieldPath": "metadata.annotations"
                }
              }],
              "defaultMode": 420
            }
          }
        }],
        "containers": [{
          "name": "seldon-container-engine",
          "image": "seldonio/engine:0.2.3",
          "workingDir": "",
          "ports": [{
            "name": "",
            "hostPort": 0,
            "containerPort": 8000,
            "protocol": "TCP",
            "hostIP": ""
          }, {
            "name": "admin",
            "hostPort": 0,
            "containerPort": 8082,
            "protocol": "TCP",
            "hostIP": ""
          }, {
            "name": "jmx",
            "hostPort": 0,
            "containerPort": 9090,
            "protocol": "TCP",
            "hostIP": ""
          }],
          "env": [{
            "name": "ENGINE_PREDICTOR",
            "value": "eyJuYW1lIjoidGVzdDEiLCJncmFwaCI6eyJuYW1lIjoidGVzdDEiLCJ0eXBlIjoiTU9ERUwiLCJlbmRwb2ludCI6eyJzZXJ2aWNlX2hvc3QiOiJ0ZXN0MS10ZXN0MS10ZXN0MSIsInNlcnZpY2VfcG9ydCI6OTAwMCwidHlwZSI6IlJFU1QifX0sImNvbXBvbmVudFNwZWNzIjpbeyJtZXRhZGF0YSI6eyJsYWJlbHMiOnsic2VsZG9uLWFwcC10ZXN0MSI6InRlc3QxLXRlc3QxLXRlc3QxIn19LCJzcGVjIjp7ImNvbnRhaW5lcnMiOlt7Im5hbWUiOiJ0ZXN0MSIsImltYWdlIjoidm14LWVlYTE2OTo1MDAwL2ltZ192Z2cxNjp2MC4wLjEiLCJwb3J0cyI6W3sibmFtZSI6Imh0dHAiLCJjb250YWluZXJQb3J0Ijo5MDAwfV0sImVudiI6W3sibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9TRVJWSUNFX1BPUlQiLCJ2YWx1ZSI6IjkwMDAifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfUEFSQU1FVEVSUyIsInZhbHVlIjoiW10ifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfSUQiLCJ2YWx1ZSI6InRlc3QxIn0seyJuYW1lIjoiUFJFRElDVE9SX0lEIiwidmFsdWUiOiJ0ZXN0MSJ9LHsibmFtZSI6IlNFTERPTl9ERVBMT1lNRU5UX0lEIiwidmFsdWUiOiJ0ZXN0MSJ9XSwidm9sdW1lTW91bnRzIjpbeyJuYW1lIjoicG9kaW5mbyIsInJlYWRPbmx5Ijp0cnVlLCJtb3VudFBhdGgiOiIvZXRjL3BvZGluZm8ifV0sImxpdmVuZXNzUHJvYmUiOnsiaGFuZGxlciI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifX0sImluaXRpYWxEZWxheVNlY29uZHMiOjEwMCwidGltZW91dFNlY29uZHMiOjEsInBlcmlvZFNlY29uZHMiOjUsInN1Y2Nlc3NUaHJlc2hvbGQiOjEsImZhaWx1cmVUaHJlc2hvbGQiOjN9LCJyZWFkaW5lc3NQcm9iZSI6eyJoYW5kbGVyIjp7InRjcFNvY2tldCI6eyJwb3J0IjoiaHR0cCJ9fSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6MTAwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDUiXX19fSwiaW1hZ2VQdWxsUG9saWN5IjoiSWZOb3RQcmVzZW50In1dLCJ0ZXJtaW5hdGlvbkdyYWNlUGVyaW9kU2Vjb25kcyI6MX19XSwicmVwbGljYXMiOjEsImFubm90YXRpb25zIjp7InByZWRpY3Rvcl92ZXJzaW9uIjoidjEifX0="
          }, {
            "name": "DEPLOYMENT_NAME",
            "value": "test1"
          }, {
            "name": "ENGINE_SERVER_PORT",
            "value": "8000"
          }, {
            "name": "ENGINE_SERVER_GRPC_PORT",
            "value": "5001"
          }, {
            "name": "JAVA_OPTS",
            "value": "-Dcom.sun.management.jmxremote.rmi.port=9090 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=127.0.0.1"
          }],
          "resources": {
            "requests": {
              "cpu": "100m"
            }
          },
          "volumeMounts": [{
            "name": "podinfo",
            "readOnly": true,
            "mountPath": "/etc/podinfo",
            "subPath": ""
          }],
          "livenessProbe": {
            "handler": {
              "httpGet": {
                "path": "/ready",
                "port": "admin",
                "host": "",
                "scheme": "HTTP"
              }
            },
            "initialDelaySeconds": 10,
            "timeoutSeconds": 2,
            "periodSeconds": 10,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "readinessProbe": {
            "handler": {
              "httpGet": {
                "path": "/ready",
                "port": "admin",
                "host": "",
                "scheme": "HTTP"
              }
            },
            "initialDelaySeconds": 10,
            "timeoutSeconds": 2,
            "periodSeconds": 10,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "lifecycle": {
            "preStop": {
              "exec": {
                "command": ["/bin/sh", "-c", "curl 127.0.0.1:8000/pause && /bin/sleep 5"]
              }
            }
          },
          "terminationMessagePath": "/dev/termination-log",
          "imagePullPolicy": "IfNotPresent",
          "securityContext": {
            "runAsUser": 8888
          },
          "stdin": false,
          "stdinOnce": false,
          "tty": false,
          "terminationMessagePolicy": "File"
        }],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 20,
        "dnsPolicy": "ClusterFirst",
        "serviceAccountName": "",
        "serviceAccount": "",
        "nodeName": "",
        "hostNetwork": false,
        "hostPID": false,
        "hostIPC": false,
        "securityContext": {
        },
        "hostname": "",
        "subdomain": "",
        "schedulerName": "default-scheduler",
        "priorityClassName": ""
      }
    },
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": "10%",
        "maxSurge": ""
      }
    },
    "minReadySeconds": 0,
    "revisionHistoryLimit": 10,
    "paused": false,
    "progressDeadlineSeconds": 600
  },
  "status": {
    "observedGeneration": 0,
    "replicas": 0,
    "updatedReplicas": 0,
    "availableReplicas": 0,
    "unavailableReplicas": 0,
    "readyReplicas": 0
  }
}
14:55:57.558 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Will try to call LIST /apis/extensions/v1beta1/namespaces/mxe/deployments/test1-test1-test1-0
14:55:57.587 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - About to CREATE {
  "metadata": {
    "name": "test1-test1-test1-0",
    "labels": {
      "seldon-deployment-id": "test1",
      "app": "test1-test1-test1-0",
      "seldon-type": "deployment",
      "seldon-app-test1": "test1-test1-test1"
    },
    "ownerReferences": [{
      "kind": "SeldonDeployment",
      "name": "test1",
      "uid": "76a9108e-3456-11e9-987f-0050569a614d",
      "apiVersion": "machinelearning.seldon.io/v1alpha2",
      "controller": true
    }]
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "seldon-app-test1": "test1-test1-test1"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "seldon-app-test1": "test1-test1-test1",
          "seldon-deployment-id": "test1"
        },
        "annotations": {
          "deployment_version": "v1",
          "project_name": "test1",
          "predictor_version": "v1"
        }
      },
      "spec": {
        "volumes": [{
          "name": "podinfo",
          "volumeSource": {
            "downwardAPI": {
              "items": [{
                "path": "annotations",
                "fieldRef": {
                  "fieldPath": "metadata.annotations"
                }
              }]
            }
          }
        }],
        "containers": [{
          "name": "test1",
          "image": "vmx-eea169:5000/img_vgg16:v0.0.1",
          "ports": [{
            "name": "http",
            "containerPort": 9000
          }],
          "env": [{
            "name": "PREDICTIVE_UNIT_SERVICE_PORT",
            "value": "9000"
          }, {
            "name": "PREDICTIVE_UNIT_PARAMETERS",
            "value": "[]"
          }, {
            "name": "PREDICTIVE_UNIT_ID",
            "value": "test1"
          }, {
            "name": "PREDICTOR_ID",
            "value": "test1"
          }, {
            "name": "SELDON_DEPLOYMENT_ID",
            "value": "test1"
          }],
          "volumeMounts": [{
            "name": "podinfo",
            "readOnly": true,
            "mountPath": "/etc/podinfo"
          }],
          "livenessProbe": {
            "handler": {
              "tcpSocket": {
                "port": "http"
              }
            },
            "initialDelaySeconds": 100,
            "timeoutSeconds": 1,
            "periodSeconds": 5,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "readinessProbe": {
            "handler": {
              "tcpSocket": {
                "port": "http"
              }
            },
            "initialDelaySeconds": 100,
            "timeoutSeconds": 1,
            "periodSeconds": 5,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "lifecycle": {
            "preStop": {
              "exec": {
                "command": ["/bin/sh", "-c", "/bin/sleep 5"]
              }
            }
          },
          "imagePullPolicy": "IfNotPresent"
        }],
        "terminationGracePeriodSeconds": 20
      }
    },
    "strategy": {
      "rollingUpdate": {
        "maxUnavailable": "10%"
      }
    }
  }
}
14:55:57.628 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Created deployment:{
  "metadata": {
    "name": "test1-test1-test1-0",
    "generateName": "",
    "namespace": "mxe",
    "selfLink": "/apis/extensions/v1beta1/namespaces/mxe/deployments/test1-test1-test1-0",
    "uid": "76c40aca-3456-11e9-987f-0050569a614d",
    "resourceVersion": "19472320",
    "generation": 1,
    "creationTimestamp": "2019-02-19T14:55:57Z",
    "labels": {
      "app": "test1-test1-test1-0",
      "seldon-app-test1": "test1-test1-test1",
      "seldon-deployment-id": "test1",
      "seldon-type": "deployment"
    },
    "ownerReferences": [{
      "kind": "SeldonDeployment",
      "name": "test1",
      "uid": "76a9108e-3456-11e9-987f-0050569a614d",
      "apiVersion": "machinelearning.seldon.io/v1alpha2",
      "controller": true
    }],
    "clusterName": ""
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "seldon-app-test1": "test1-test1-test1"
      }
    },
    "template": {
      "metadata": {
        "name": "",
        "generateName": "",
        "namespace": "",
        "selfLink": "",
        "uid": "",
        "resourceVersion": "",
        "generation": 0,
        "creationTimestamp": "1970-01-01T00:00:00Z",
        "labels": {
          "seldon-app-test1": "test1-test1-test1",
          "seldon-deployment-id": "test1"
        },
        "annotations": {
          "deployment_version": "v1",
          "predictor_version": "v1",
          "project_name": "test1"
        },
        "clusterName": ""
      },
      "spec": {
        "volumes": [{
          "name": "podinfo",
          "volumeSource": {
            "downwardAPI": {
              "items": [{
                "path": "annotations",
                "fieldRef": {
                  "apiVersion": "v1",
                  "fieldPath": "metadata.annotations"
                }
              }],
              "defaultMode": 420
            }
          }
        }],
        "containers": [{
          "name": "test1",
          "image": "vmx-eea169:5000/img_vgg16:v0.0.1",
          "workingDir": "",
          "ports": [{
            "name": "http",
            "hostPort": 0,
            "containerPort": 9000,
            "protocol": "TCP",
            "hostIP": ""
          }],
          "env": [{
            "name": "PREDICTIVE_UNIT_SERVICE_PORT",
            "value": "9000"
          }, {
            "name": "PREDICTIVE_UNIT_PARAMETERS",
            "value": "[]"
          }, {
            "name": "PREDICTIVE_UNIT_ID",
            "value": "test1"
          }, {
            "name": "PREDICTOR_ID",
            "value": "test1"
          }, {
            "name": "SELDON_DEPLOYMENT_ID",
            "value": "test1"
          }],
          "resources": {
          },
          "volumeMounts": [{
            "name": "podinfo",
            "readOnly": true,
            "mountPath": "/etc/podinfo",
            "subPath": ""
          }],
          "livenessProbe": {
            "handler": {
              "tcpSocket": {
                "port": "http",
                "host": ""
              }
            },
            "initialDelaySeconds": 100,
            "timeoutSeconds": 1,
            "periodSeconds": 5,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "readinessProbe": {
            "handler": {
              "tcpSocket": {
                "port": "http",
                "host": ""
              }
            },
            "initialDelaySeconds": 100,
            "timeoutSeconds": 1,
            "periodSeconds": 5,
            "successThreshold": 1,
            "failureThreshold": 3
          },
          "lifecycle": {
            "preStop": {
              "exec": {
                "command": ["/bin/sh", "-c", "/bin/sleep 5"]
              }
            }
          },
          "terminationMessagePath": "/dev/termination-log",
          "imagePullPolicy": "IfNotPresent",
          "stdin": false,
          "stdinOnce": false,
          "tty": false,
          "terminationMessagePolicy": "File"
        }],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 20,
        "dnsPolicy": "ClusterFirst",
        "serviceAccountName": "",
        "serviceAccount": "",
        "nodeName": "",
        "hostNetwork": false,
        "hostPID": false,
        "hostIPC": false,
        "securityContext": {
        },
        "hostname": "",
        "subdomain": "",
        "schedulerName": "default-scheduler",
        "priorityClassName": ""
      }
    },
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": "10%",
        "maxSurge": ""
      }
    },
    "minReadySeconds": 0,
    "revisionHistoryLimit": 10,
    "paused": false,
    "progressDeadlineSeconds": 600
  },
  "status": {
    "observedGeneration": 0,
    "replicas": 0,
    "updatedReplicas": 0,
    "availableReplicas": 0,
    "unavailableReplicas": 0,
    "readyReplicas": 0
  }
}
14:55:57.975 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Created service:{
  "metadata": {
    "name": "test1-test1-test1",
    "generateName": "",
    "namespace": "mxe",
    "selfLink": "/api/v1/namespaces/mxe/services/test1-test1-test1",
    "uid": "76fca816-3456-11e9-987f-0050569a614d",
    "resourceVersion": "19472344",
    "generation": 0,
    "creationTimestamp": "2019-02-19T14:55:57Z",
    "labels": {
      "seldon-app-test1": "test1-test1-test1",
      "seldon-deployment-id": "test1"
    },
    "ownerReferences": [{
      "kind": "SeldonDeployment",
      "name": "test1",
      "uid": "76a9108e-3456-11e9-987f-0050569a614d",
      "apiVersion": "machinelearning.seldon.io/v1alpha2",
      "controller": true
    }],
    "clusterName": ""
  },
  "spec": {
    "ports": [{
      "name": "http",
      "protocol": "TCP",
      "port": 9000,
      "targetPort": "",
      "nodePort": 0
    }],
    "selector": {
      "seldon-app-test1": "test1-test1-test1"
    },
    "clusterIP": "10.233.33.105",
    "type": "ClusterIP",
    "sessionAffinity": "None",
    "loadBalancerIP": "",
    "externalName": "",
    "externalTrafficPolicy": "",
    "healthCheckNodePort": 0
  },
  "status": {
    "loadBalancer": {
    }
  }
}
14:55:58.419 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Created service:{
  "metadata": {
    "name": "test1",
    "generateName": "",
    "namespace": "mxe",
    "selfLink": "/api/v1/namespaces/mxe/services/test1",
    "uid": "773f799a-3456-11e9-987f-0050569a614d",
    "resourceVersion": "19472351",
    "generation": 0,
    "creationTimestamp": "2019-02-19T14:55:58Z",
    "labels": {
      "seldon-app": "test1",
      "seldon-deployment-id": "test1"
    },
    "annotations": {
      "getambassador.io/config": "---\napiVersion: ambassador/v0\nkind:  Mapping\nname:  seldon_test1_rest_mapping\nprefix: /seldon/test1/\nservice: test1:8000\n---\napiVersion: ambassador/v0\nkind:  Mapping\nname:  test1_grpc_mapping\ngrpc: true\nprefix: /seldon.protos.Seldon/\nrewrite: /seldon.protos.Seldon/\nheaders:\n  seldon: test1\nservice: test1:5001\n"
    },
    "ownerReferences": [{
      "kind": "SeldonDeployment",
      "name": "test1",
      "uid": "76a9108e-3456-11e9-987f-0050569a614d",
      "apiVersion": "machinelearning.seldon.io/v1alpha2",
      "controller": true
    }],
    "clusterName": ""
  },
  "spec": {
    "ports": [{
      "name": "http",
      "protocol": "TCP",
      "port": 8000,
      "targetPort": "",
      "nodePort": 0
    }, {
      "name": "grpc",
      "protocol": "TCP",
      "port": 5001,
      "targetPort": "",
      "nodePort": 0
    }],
    "selector": {
      "seldon-app": "test1"
    },
    "clusterIP": "10.233.11.10",
    "type": "ClusterIP",
    "sessionAffinity": "None",
    "loadBalancerIP": "",
    "externalName": "",
    "externalTrafficPolicy": "",
    "healthCheckNodePort": 0
  },
  "status": {
    "loadBalancer": {
    }
  }
}
14:55:58.493 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Pushing updated SeldonDeployment test1 back to kubectl
14:55:58.494 [pool-1-thread-1] DEBUG i.s.c.k8s.KubeCRDHandlerImpl - Updating seldondeployment test1
14:55:58.714 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - MODIFIED
 : {"apiVersion":"machinelearning.seldon.io/v1alpha2","kind":"SeldonDeployment","metadata":{"annotations":{"ksonnet.io/managed":"{\"pristine\":\"H4sIAAAAAAAA/8yS0U7dPAzH77/H8HW/Q7sJpPUaNE0a29GKdjMhZBLTeiR2lKQVR6jvPqWFjqPDA+wusePf33/Hz4CBf1JMrAIteDQDCznCKCz9LpGzKjvWs6lBFwb8ABU8slhooVtylxScHjxJhgo8ZbSYEdpncHhPLpUThgAtrKhSnlSEcoEa9UGl1LaQKeUG5goEPW339ZoCmhLzT1RepEBmAYtoxswqi47dWrmbNkdTYYSov8nkuyP0qVSIZNlkjQnaXyf4LXtMnyvYbHSBzFr72qJRychCL0j22BfFyT/9T4TNxaf2vK7rM/b93dT3zUU71bt6V5pZnu5H5/bq2ByghS8P3zTvI6V12I4nEkppH/WeitYDshsj3QyR0qDOQvuxggHFOooln03o1DxSXtxoLFMfcg4wzxWwcGZ0l+Tw0JFRsQnapq4rCBRZ7RY7ryCNxlBKb4SaCjJ70jH/rT2dbyS0/I+3PKkbPV3rKLl82e18++YrOjKR1ngFmaJnWfbjc0RD+2PR5pW1YgqnjxiGZSsGdjaSrCASG5Rl8ZgPoUzsx1V3886CvmSvv19efX0nHSk4Nrg4uZ3n//4AAAD//wEAAP//IkEGlN0DAAA\u003d\"}","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"machinelearning.seldon.io/v1alpha2\",\"kind\":\"SeldonDeployment\",\"metadata\":{\"name\":\"test1\",\"namespace\":\"mxe\",\"selfLink\":\"/apis/machinelearning.seldon.io/v1alpha2/namespaces/mxe/seldondeployments/test1\",\"uid\":\"76a9108e-3456-11e9-987f-0050569a614d\",\"generation\":1,\"creationTimestamp\":\"2019-02-19T14:55:57Z\",\"labels\":{\"app\":\"seldon\",\"app.kubernetes.io/deploy-manager\":\"ksonnet\",\"ksonnet.io/component\":\"test1\"},\"annotations\":{\"ksonnet.io/managed\":\"{\\\"pristine\\\":\\\"H4sIAAAAAAAA/8yS0U7dPAzH77/H8HW/Q7sJpPUaNE0a29GKdjMhZBLTeiR2lKQVR6jvPqWFjqPDA+wusePf33/Hz4CBf1JMrAIteDQDCznCKCz9LpGzKjvWs6lBFwb8ABU8slhooVtylxScHjxJhgo8ZbSYEdpncHhPLpUThgAtrKhSnlSEcoEa9UGl1LaQKeUG5goEPW339ZoCmhLzT1RepEBmAYtoxswqi47dWrmbNkdTYYSov8nkuyP0qVSIZNlkjQnaXyf4LXtMnyvYbHSBzFr72qJRychCL0j22BfFyT/9T4TNxaf2vK7rM/b93dT3zUU71bt6V5pZnu5H5/bq2ByghS8P3zTvI6V12I4nEkppH/WeitYDshsj3QyR0qDOQvuxggHFOooln03o1DxSXtxoLFMfcg4wzxWwcGZ0l+Tw0JFRsQnapq4rCBRZ7RY7ryCNxlBKb4SaCjJ70jH/rT2dbyS0/I+3PKkbPV3rKLl82e18++YrOjKR1ngFmaJnWfbjc0RD+2PR5pW1YgqnjxiGZSsGdjaSrCASG5Rl8ZgPoUzsx1V3886CvmSvv19efX0nHSk4Nrg4uZ3n//4AAAD//wEAAP//IkEGlN0DAAA\u003d\\\"}\"},\"clusterName\":\"\"},\"spec\":{\"name\":\"test1\",\"predictors\":[{\"name\":\"test1\",\"graph\":{\"name\":\"test1\",\"type\":\"MODEL\",\"endpoint\":{\"service_host\":\"test1-test1-test1\",\"service_port\":9000,\"type\":\"REST\"}},\"componentSpecs\":[{\"metadata\":{\"labels\":{\"seldon-app-test1\":\"test1-test1-test1\"}},\"spec\":{\"containers\":[{\"name\":\"test1\",\"image\":\"vmx-eea169:5000/img_vgg16:v0.0.1\",\"ports\":[{\"name\":\"http\",\"containerPort\":9000}],\"env\":[{\"name\":\"PREDICTIVE_UNIT_SERVICE_PORT\",\"value\":\"9000\"},{\"name\":\"PREDICTIVE_UNIT_PARAMETERS\",\"value\":\"[]\"},{\"name\":\"PREDICTIVE_UNIT_ID\",\"value\":\"test1\"},{\"name\":\"PREDICTOR_ID\",\"value\":\"test1\"},{\"name\":\"SELDON_DEPLOYMENT_ID\",\"value\":\"test1\"}],\"volumeMounts\":[{\"name\":\"podinfo\",\"readOnly\":true,\"mountPath\":\"/etc/podinfo\"}],\"livenessProbe\":{\"handler\":{\"tcpSocket\":{\"port\":\"http\"}},\"initialDelaySeconds\":100,\"timeoutSeconds\":1,\"periodSeconds\":5,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"handler\":{\"tcpSocket\":{\"port\":\"http\"}},\"initialDelaySeconds\":100,\"timeoutSeconds\":1,\"periodSeconds\":5,\"successThreshold\":1,\"failureThreshold\":3},\"lifecycle\":{\"preStop\":{\"exec\":{\"command\":[\"/bin/sh\",\"-c\",\"/bin/sleep 5\"]}}},\"imagePullPolicy\":\"IfNotPresent\"}],\"terminationGracePeriodSeconds\":1}}],\"replicas\":1,\"annotations\":{\"predictor_version\":\"v1\"}}],\"annotations\":{\"deployment_version\":\"v1\",\"project_name\":\"test1\"}},\"status\":{\"state\":\"Creating\"}}\n"},"clusterName":"","creationTimestamp":"2019-02-19T14:55:57Z","generation":1.0,"labels":{"app":"seldon","app.kubernetes.io/deploy-manager":"ksonnet","ksonnet.io/component":"test1"},"name":"test1","namespace":"mxe","resourceVersion":"19472355","selfLink":"/apis/machinelearning.seldon.io/v1alpha2/namespaces/mxe/seldondeployments/test1","uid":"76a9108e-3456-11e9-987f-0050569a614d"},"spec":{"annotations":{"deployment_version":"v1","project_name":"test1"},"name":"test1","predictors":[{"annotations":{"predictor_version":"v1"},"componentSpecs":[{"metadata":{"labels":{"seldon-app-test1":"test1-test1-test1"}},"spec":{"containers":[{"env":[{"name":"PREDICTIVE_UNIT_SERVICE_PORT","value":"9000"},{"name":"PREDICTIVE_UNIT_PARAMETERS","value":"[]"},{"name":"PREDICTIVE_UNIT_ID","value":"test1"},{"name":"PREDICTOR_ID","value":"test1"},{"name":"SELDON_DEPLOYMENT_ID","value":"test1"}],"image":"vmx-eea169:5000/img_vgg16:v0.0.1","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","/bin/sleep 5"]}}},"livenessProbe":{"failureThreshold":3.0,"handler":{"tcpSocket":{"port":"http"}},"initialDelaySeconds":100.0,"periodSeconds":5.0,"successThreshold":1.0,"timeoutSeconds":1.0},"name":"test1","ports":[{"containerPort":9000.0,"name":"http"}],"readinessProbe":{"failureThreshold":3.0,"handler":{"tcpSocket":{"port":"http"}},"initialDelaySeconds":100.0,"periodSeconds":5.0,"successThreshold":1.0,"timeoutSeconds":1.0},"volumeMounts":[{"mountPath":"/etc/podinfo","name":"podinfo","readOnly":true}]}],"terminationGracePeriodSeconds":1.0}}],"graph":{"endpoint":{"service_host":"test1-test1-test1","service_port":9000.0,"type":"REST"},"name":"test1","type":"MODEL"},"name":"test1","replicas":1.0}]},"status":{"state":"Creating"}}

14:55:58.715 [pool-1-thread-1] DEBUG i.s.c.k.SeldonDeploymentControllerImpl - Only updated cache for test1
14:56:08.716 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Updating processed resource version to 19472355
14:56:08.717 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:56:08
14:56:08.717 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:56:12.818 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:56:13.717 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:56:13
14:56:13.717 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 19472355 in namespace mxe
14:56:23.763 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:19472355 Processed:19472355
14:56:23.763 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:56:23
14:56:23.763 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:56:24.156 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:56:28.764 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:56:28
14:56:28.764 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 19472355 in namespace mxe
14:56:38.810 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:19472355 Processed:19472355
14:56:38.810 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:56:38
14:56:38.810 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:56:38.871 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:56:43.810 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:56:43
14:56:43.811 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 19472355 in namespace mxe
14:56:53.858 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:19472355 Processed:19472355
14:56:53.858 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:56:53
14:56:53.858 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:56:53.918 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:56:58.858 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:56:58
14:56:58.858 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 19472355 in namespace mxe
14:57:08.904 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:19472355 Processed:19472355
14:57:08.904 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:57:08
14:57:08.904 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:57:08.964 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:57:13.905 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:57:13
14:57:13.905 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 19472355 in namespace mxe
14:57:23.953 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:19472355 Processed:19472355
14:57:23.953 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:57:23
14:57:23.953 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:57:24.020 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:57:28.953 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:57:28
14:57:28.953 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 19472355 in namespace mxe
14:57:39.002 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:19472355 Processed:19472355
14:57:39.002 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:57:39
14:57:39.002 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:57:39.058 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
14:57:44.002 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - The time is now 14:57:44
14:57:44.002 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Watching with rs 19472355 in namespace mxe
14:57:54.050 [pool-1-thread-1] DEBUG i.s.c.k8s.SeldonDeploymentWatcher - Not updating resourceVersion - current:19472355 Processed:19472355
14:57:54.050 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - The time is now 14:57:54
14:57:54.051 [pool-1-thread-1] DEBUG i.s.c.k8s.DeploymentWatcher - Watching with rs 1016583
14:57:54.107 [pool-1-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.lang.NullPointerException: null
  • The basic AB test component is an internal component so there is no external endpoint.

Then what is the generic way to determine from a SeldonSeployment what type endpoint it is using? (REST or GRPC?)
Or is this only for the model containers that can be either REST or GRPC? Once I deploy it as a SeldonDeployment, will the endpoint always be REST?

@ukclivecox
Copy link
Contributor

Seldon will always provision REST and gRPC endpoints externally.
Internally each component can be either REST or gRPC depending on how they are wrapped.

@ukclivecox
Copy link
Contributor

For the issue on creating status it would be great if you could try a more recent release of seldon - 0.2.4 or 0.2.5?

@sasvaritoni
Copy link
Contributor Author

OK, in a few days I will be able to update my cluster, and then retest this.

@sasvaritoni
Copy link
Contributor Author

Have not seen this issue (stuck in creating) anymore with later versions.
Ticket should be closed I think, sorry for the late answer...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ukclivecox @sasvaritoni