consul_autopilot_healthy is reporting zero on the leader

[arielazem]

Overview of the Issue
With Consul v1.10.1 (Revision db839f1), the metric consul_autopilot_healthy is zero for all the servers including the leader, even when autopilot reports the cluster is healthy

Zero on all the servers

The autopilot health endpoint reports the cluster is healthy.

Reproduction Steps
Steps to reproduce this issue, eg:

Run Consul v1.10.1
Go to /v1/agent/metrics?format=prometheus
Operating system and Environment details
Linux VM in GCP

[mikemorris]

Dupe of #10730, or is this distinct?

[arielazem]

@mikemorris , #10730 is about all the followers reporting 0, while this other ticket #11152 is about leader reporting 0 as well, although things might be related, it's a different issue

[acpana]

hey @arielazem thanks for opening this issue with us!

I actually do believe that the underlying cause is a change in one of our dependencies (go-metrics). The behavior gets manifested as metrics that were previously NaN (whether in a certain stage of leadership or not) showing up with a 0 value.

As the team works on a fix, we'll track progress on #10730 for now so I will close this issue as a duplicate. But please feel free to re-open as needed.

---

For anyone that's coming after, I added some more descriptive steps on how to repro the problem with the metrics here:

#10730 (comment)