Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd_debugging_mvcc_keys_total inconsistency #8388

Closed
mwf opened this issue Aug 10, 2017 · 3 comments
Closed

etcd_debugging_mvcc_keys_total inconsistency #8388

mwf opened this issue Aug 10, 2017 · 3 comments

Comments

@mwf
Copy link

mwf commented Aug 10, 2017

This issue is connected with #8387

After the cluster stabilized we have inconsistency in "etcd_debugging_mvcc_keys_total" metric between hosts:

4

curl http://<host3>:2379/metrics | grep etcd_debugging_mvcc_keys_total  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  3 35906    3  1328    0     0   1708      0  0:00:21 --:--:--  0:00:21  1706# HELP etcd_debugging_mvcc_keys_total Total number of keys.
# TYPE etcd_debugging_mvcc_keys_total gauge
etcd_debugging_mvcc_keys_total 15007
100 35906  100 35906    0     0  30216      0  0:00:01  0:00:01 --:--:-- 30223

One host reports it has 15007 keys.

Though range request ETCDCTL_API=3 etcdctl --endpoints="http://<host3>:2379" get --prefix --keys-only --limit=1 -w=json "" tells it has only "count":7671 keys, which is the same as other hosts report in metrics.

Maybe this issue was already fixed in newer etcd server versions? I just decided to report.

@mwf
Copy link
Author

mwf commented Aug 10, 2017

Maybe it's somehow connected with my old issue #8146, which is fixed now.

@heyitsanthony
Copy link
Contributor

@mwf can this be reproduced with master?

@mwf
Copy link
Author

mwf commented Aug 10, 2017

Actually, I did not find a way to reproduce it, the issue is too strange to be true, but it's a fact :)

I'll try, if I manage to reproduce I will let you know. But if you could take a look at code producing this metric value and maybe find a bug - that would be great. Just as you did in #8146.

heyitsanthony pushed a commit to heyitsanthony/etcd that referenced this issue Aug 10, 2017
gyuho pushed a commit that referenced this issue Aug 10, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants