Etcd failed to open database with error Invalid database on s390x machine

[mitiao]

We have a running Kubernetes cluster with one x86 master node, and when add/expand a s390x master node into this cluster, the etcd got panic error with invalid database:

Aug 07 06:19:30 s390_host etcd[168270]: {"level":"panic","ts":"2024-08-07T06:19:30.873Z","caller":"backend/backend.go:189","msg":"failed to open database","path":"/var/lib/etcd/member/snap/db","error":"invalid database","stacktrace":"go.etcd.io/etcd/server/v3/mvcc/backend.newBackend\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:189\ngo.etcd.io/etcd/server/v3/mvcc/backend.New\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:163\ngo.etcd.io/etcd/server/v3/etcdserver.newBackend\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:55\ngo.etcd.io/etcd/server/v3/etcdserver.openBackend.func1\n\t/tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:76"}
Aug 07 06:19:30 s390_host etcd[168270]: panic: failed to open database
Aug 07 06:19:30 s390_host etcd[168270]: goroutine 63 [running]:
Aug 07 06:19:30 s390_host etcd[168270]: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000470000, 0xc0000fac00, 0x2, 0x2)
Aug 07 06:19:30 s390_host etcd[168270]:         /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:234 +0x446
Aug 07 06:19:30 s390_host etcd[168270]: go.uber.org/zap.(*Logger).Panic(0xc000458000, 0xd78520, 0x17, 0xc0000fac00, 0x2, 0x2)
Aug 07 06:19:30 s390_host etcd[168270]:         /usr/local/google/home/siarkowicz/.gvm/pkgsets/go1.16.15/global/pkg/mod/go.uber.org/[email protected]/logger.go:227 +0x76
Aug 07 06:19:30 s390_host etcd[168270]: go.etcd.io/etcd/server/v3/mvcc/backend.newBackend(0xc000045ae0, 0x1c, 0x5f5e100, 0x2710, 0xd6707c, 0x7, 0x233333333, 0xc000458000, 0x0, 0xee9e88, ...)
Aug 07 06:19:30 s390_host etcd[168270]:         /tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:189 +0x21c
Aug 07 06:19:30 s390_host etcd[168270]: go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
Aug 07 06:19:30 s390_host etcd[168270]:         /tmp/etcd-release-3.5.5/etcd/release/etcd/server/mvcc/backend/backend.go:163
Aug 07 06:19:30 s390_host etcd[168270]: go.etcd.io/etcd/server/v3/etcdserver.newBackend(0x3fff13ffa20, 0x16, 0x0, 0x0, 0x0, 0x0, 0xc0002b1ef0, 0x1, 0x1, 0xc0004d8750, ...)
Aug 07 06:19:30 s390_host etcd[168270]:         /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:55 +0x158
Aug 07 06:19:30 s390_host etcd[168270]: go.etcd.io/etcd/server/v3/etcdserver.openBackend.func1(0xc00045c720, 0xc0000c4900, 0xee9e88, 0xc0004d8630)
Aug 07 06:19:30 s390_host etcd[168270]:         /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:76 +0x68
Aug 07 06:19:30 s390_host etcd[168270]: created by go.etcd.io/etcd/server/v3/etcdserver.openBackend
Aug 07 06:19:30 s390_host etcd[168270]:         /tmp/etcd-release-3.5.5/etcd/release/etcd/server/etcdserver/backend.go:75 +0x112

both of x86 and s390x master node use same version etcd 3.5.5
We had some investigation and found that the etcd database was created by x86 machine that can not be opened on s390x machine, vice versa.

We did a test on both of x86 and s390x to use bbolt to check the database created by x86 and s390x:
on s390x machine:

$ bbolt check z_master_db
OK
$ bbolt check x_master_db
invalid database
$ bbolt page -all --format-value=redacted x_master_db
cannot read number of pages: the Meta Page has wrong (unexpected) magic

on x86 machine:

$ bbolt check x_master_db
OK
$ bbolt check z_master_db
invalid database
$ bbolt page -all --format-value=redacted z_master_db
cannot read number of pages: the Meta Page has wrong (unexpected) magic

Question:
Is the bbolt db file arch-specific?
Any solution to solve this issue?
Thanks!

[ahrtr]

It's expected behaviour. You can't move db file from one platform to another platform, because different platforms may have different pageSize and endian. You can compare the differences using https://github.com/ahrtr/systeminfo.

Building a cluster using machines from different platforms might be feasible in the beginning. But it will be problematic once sending a snapshot from the leader to a follower with a different platform. But of course, I do not have such environment to test it out.

So the suggest for now is to always use machines with the same systeminfo (pageSize and endian) to run an etcd cluster.

[mitiao]

Thanks for your answer!