Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

after upgrade nebula-operator version, operator start failed #229

Closed
jinyingsunny opened this issue Jul 31, 2023 · 2 comments
Closed

after upgrade nebula-operator version, operator start failed #229

jinyingsunny opened this issue Jul 31, 2023 · 2 comments
Labels
affects/none PR/issue: this bug affects none version. process/fixed Process of bug severity/none Severity of bug type/bug Type: something is unexpected

Comments

@jinyingsunny
Copy link

gcp environment, harris has created a nebula cluster on vesoft/nebula-operator:v1.4.2 ;
after upgrate operator version to reg.vesoft-inc.com/cloud-dev/nebula-operator:snap-1.4 ;

operator start failed

$ kubectl get pod -n nebula-operator-system
NAME                                                             READY   STATUS             RESTARTS        AGE
nebula-operator-controller-manager-deployment-7985874b6d-tsjgp   1/2     CrashLoopBackOff   8 (3m48s ago)   22m

logs:

$ kubectl logs nebula-operator-controller-manager-deployment-7985874b6d-tsjgp  -n nebula-operator-system
Defaulted container "controller-manager" out of: controller-manager, kube-rbac-proxy
I0731 06:52:43.774391       1 main.go:123] setup "msg"="Welcome to Nebula Operator."
I0731 06:52:43.774544       1 main.go:124] setup "msg"="Nebula Operator Version" "version"={"gitVersion":"6290403281110773-dev","gitCommit":"62904032811107735eb3147e7189b105a1374702","gitDate":"2023-05-31T08:16:46Z","buildDate":"2023-07-27T11:21:11Z","goVersion":"go1.19.7","compiler":"gc","platform":"linux/amd64"}
I0731 06:52:44.468606       1 listener.go:44] controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"=":8080"
I0731 06:52:45.779061       1 main.go:185] setup "msg"="starting manager"
I0731 06:52:45.779684       1 internal.go:369]  "msg"="Starting server" "addr"={"IP":"::","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
I0731 06:52:45.779729       1 internal.go:369]  "msg"="Starting server" "addr"={"IP":"::","Port":8081,"Zone":""} "kind"="health probe"
I0731 06:52:45.779792       1 leaderelection.go:248] attempting to acquire leader lease nebula-operator-system/nebula-controller-manager-leader...
I0731 06:53:02.839277       1 leaderelection.go:258] successfully acquired lease nebula-operator-system/nebula-controller-manager-leader
I0731 06:53:02.839674       1 controller.go:186]  "msg"="Starting EventSource" "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster" "source"="kind source: *v1alpha1.NebulaCluster"
I0731 06:53:02.839722       1 controller.go:186]  "msg"="Starting EventSource" "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster" "source"="kind source: *v1.ConfigMap"
I0731 06:53:02.839746       1 controller.go:186]  "msg"="Starting EventSource" "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster" "source"="kind source: *v1.Service"
I0731 06:53:02.839767       1 controller.go:186]  "msg"="Starting EventSource" "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster" "source"="kind source: *v1.StatefulSet"
I0731 06:53:02.839790       1 controller.go:186]  "msg"="Starting EventSource" "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster" "source"="kind source: *v1.Deployment"
I0731 06:53:02.839812       1 controller.go:194]  "msg"="Starting Controller" "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster"
I0731 06:53:02.839873       1 controller.go:186]  "msg"="Starting EventSource" "controller"="nebularestore" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaRestore" "source"="kind source: *v1alpha1.NebulaRestore"
I0731 06:53:02.839902       1 controller.go:194]  "msg"="Starting Controller" "controller"="nebularestore" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaRestore"
I0731 06:53:02.941119       1 controller.go:228]  "msg"="Starting workers" "controller"="nebularestore" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaRestore" "worker count"=5
I0731 06:53:02.941825       1 controller.go:228]  "msg"="Starting workers" "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster" "worker count"=3
I0731 06:53:02.942308       1 nebula_cluster_controller.go:160] Start to reconcile NebulaCluster
I0731 06:53:03.617499       1 nebula_cluster_controller.go:143] Finished reconciling NebulaCluster [harris/root], spendTime: (675.407398ms)
I0731 06:53:03.617838       1 controller.go:118]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "NebulaCluster"={"name":"root","namespace":"harris"} "controller"="nebulacluster" "controllerGroup"="apps.nebula-graph.io" "controllerKind"="NebulaCluster" "name"="root" "namespace"="harris" "reconcileID"="856655c4-0aaa-4158-82af-7dcaf6986454"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x14a3141]

reason is nebula-cluster with logRotate open without set agent image and version。
but before upgrade ,it is ok.

Describe the bug (required)

upgrade normal

Your Environments (required)
nebula-ent-sc-rc包
镜像是:
image: reg.vesoft-inc.com/rc/nebula-storaged-ent
version: v3.5.0-sc
operator镜像是:
reg.vesoft-inc.com/cloud-dev/nebula-operator:snap-1.4

@jinyingsunny jinyingsunny added the type/bug Type: something is unexpected label Jul 31, 2023
@github-actions github-actions bot added affects/none PR/issue: this bug affects none version. severity/none Severity of bug labels Jul 31, 2023
@MegaByte875
Copy link
Contributor

#232

@jinyingsunny
Copy link
Author

checked.

@github-actions github-actions bot added the process/fixed Process of bug label Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects/none PR/issue: this bug affects none version. process/fixed Process of bug severity/none Severity of bug type/bug Type: something is unexpected
Projects
None yet
Development

No branches or pull requests

2 participants