Using multiple copies, master-slave asynchronous replication storage, there are inaccurate update metrics data, such as MySQL and Clickhouse

[sglztc]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

Hello, Mr. Wu, the oap metrics updates the metrics every 25 seconds. The update process is to query the latest historical metrics data first, then aggregate it with the current value, and then save it to the storage. If the current storage uses multiple copies and the master-slave replication is asynchronous, there will be inaccurate metrics updates. For example, during asynchronous replication, the data has not been copied to the slave node, and the slave node is queried at this time. At this time, the query data is inaccurate, resulting in inaccurate update metrics data. For example, if MySQL uses master-slave or Clickhouse uses multiple copies, this problem will occur. Hello, Mr. Wu, how to deal with this problem?

What you expected to happen

Hello, Mr. Wu, the oap metrics updates the metrics every 25 seconds. The update process is to query the latest historical metrics data first, then aggregate it with the current value, and then save it to the storage. If the current storage uses multiple copies and the master-slave replication is asynchronous, there will be inaccurate metrics updates. For example, during asynchronous replication, the data has not been copied to the slave node, and the slave node is queried at this time. At this time, the query data is inaccurate, resulting in inaccurate update metrics data. For example, if MySQL uses master-slave or Clickhouse uses multiple copies, this problem will occur. Hello, Mr. Wu, how to deal with this problem?

How to reproduce

When the master-slave data is not synchronized, such as ClickHouse, we shut down the coordinator keeper. At this time, the master-slave will not be synchronized, and the data on both sides will be inconsistent.

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Hello, Mr. Wu, the oap metrics updates the metrics every 25 seconds. The update process is to query the latest historical metrics data first, then aggregate it with the current value, and then save it to the storage.

How do you get the conclusion? OAP doesn't read, it has internal cache.

For example, if MySQL uses master-slave or Clickhouse uses multiple copies, this problem will occur.

We don't have Clickhouse storage option. This is not our concern.

[wu-sheng]

Be clear, APLv2 provides you all rights to change codes to run in any way you like. But we wouldn't review your private changes, and can't support that.

[sglztc]

Hello, Mr. Wu, we just turned off the switch of enableDatabaseSession, and did not change other codes. It has nothing to do with whether to read from the cache, but if the slave node data is read, the updated metrics may not be accurate because the current storage master-slave synchronization is asynchronous.

[wu-sheng]

We don't suggest you to do that as you are reading data fron an async node.
We never expected this use case.

[wu-sheng]

This is your MySQL limitation, not SkyWalking's. There is no magic to change that.

[sglztc]

Thanks