seldon-controller-manager crashing #2066
Comments
|
Did this occur when you deployed a particular model? it maybe be an issue that the webhooks are not being called. |
|
can you show the yaml of your resource? You could also try changing the version of the SeldonDeployment to v1 |
|
My resource definition for the model is already the following : I am not sure but I think this happened when I tried to deploy a model. After that, I tried restarting seldon-manager-controller, I started getting the following errors for old seldon deployments and it gave segmentation error again: |
|
It looks like its not calling the webhook for the model. Have you followed: https://www.kubeflow.org/docs/components/serving/seldon/ In particualar fo ryou rnamespace done: |
|
Before this issue, I was able to deploy the test deployment given in the doc. Should I re-deploy seldon? |
|
Ok, so the issue is solved now. My setup is such that kubeflow is there in 'kubeflow' ns and seldon deploys in 'models' ns, but once I had tried deploying seldon model in 'kubeflow' ns. I deleted that sdep and restarted, it worked. However, this is just my theory. @cliveseldon you may close this issue if you think this is correct. Thanks a lot for your help! |
|
Yes you can't deploy in kubeflow namespace. |
Describe the bug
We are running seldon with kubeflow in GCP. It was working fine but since 2 days it is crashing. The pod is going crashbackloop.
Logs from seldon-controller manger:
`E0702 17:48:38.000449 1 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:76
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:65
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/workspace/controllers/seldondeployment_prepackaged_servers.go:255
/workspace/controllers/seldondeployment_prepackaged_servers.go:255
/workspace/controllers/seldondeployment_controller.go:452
/workspace/controllers/seldondeployment_controller.go:1350
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x11bcfcb]
goroutine 358 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:58 +0x105
panic(0x130e6a0, 0x21c4930)
/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/seldonio/seldon-core/operator/apis/machinelearning/v1.IsPrepack(...)
/workspace/apis/machinelearning/v1/seldondeployment_webhook.go:125
github.com/seldonio/seldon-core/operator/controllers.createStandaloneModelServers(0xc0000d6180, 0xc000b23860, 0xc000d6a6c0, 0xc000978300, 0xc000296fa0, 0xc0001232b0, 0x0)
/workspace/controllers/seldondeployment_prepackaged_servers.go:255 +0x1cb
github.com/seldonio/seldon-core/operator/controllers.createComponents(0xc0000d6180, 0xc000b23860, 0x1698ce0, 0xc00092d5a0, 0x21f7760, 0xc000902f50, 0x8)
/workspace/controllers/seldondeployment_controller.go:452 +0x15a4
github.com/seldonio/seldon-core/operator/controllers.(*SeldonDeploymentReconciler).Reconcile(0xc0000d6180, 0xc000902f50, 0x8, 0xc000882080, 0x16, 0x21d8ce0, 0x42bd21, 0x166c700, 0xc000b21d88)
/workspace/controllers/seldondeployment_controller.go:1350 +0x4ae
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00068e000, 0x1358a80, 0xc00050c0c0, 0x1358a00)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x149
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00068e000, 0xc0004d0800)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:192 +0xb5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00068e000)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:171 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0003c44d0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:152 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003c44d0, 0x3b9aca00, 0x0, 0xc000260f01, 0xc000020000)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0003c44d0, 0x3b9aca00, 0xc000020000)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:157 +0x311`
To reproduce
Expected behaviour
The model was supposed to get deployed with seldon-core-microservice
Environment
kubectl version]:kubectl get --namespace seldon-system deploy seldon-controller-manager -o yaml | grep seldonio]:The text was updated successfully, but these errors were encountered: