[receiver/mysqlreceiver] Mysql receiver - Uint64 to Int64 overflow

[dloucasfx]

Component(s)

receiver/mysql

What happened?

Description

User is hitting this error:
2024-09-12T10:05:08.788Z error scraperhelper/scrapercontroller.go:197 Error scraping metrics {"kind": "receiver", "name": "mysql/replica", "data_type": "metrics", "error": "sql: Scan error on column index 7, name "SUM_TIMER_FETCH": converting driver.Value type uint64 ("10607806269779284266") to a int64: value out of range; sql: Scan error on column index 7, name "SUM_TIMER_FETCH": converting driver.Value type uint64 ("10607806392347803736") to a int64: value out of range; failed to parse int64 for MysqlBufferPoolPages, value was 18446744073709551264: strconv.ParseInt: parsing "18446744073709551264": value out of range", "scraper": "mysql"}

Analysis

• For SUM_TIMER_FETCH we can safely avoid the overflow, by setting timeFetch to uint64 and convert it back to int64 after the division

  opentelemetry-collector-contrib/receiver/mysqlreceiver/scraper.go

  Lines 456 to 458 in 1d12566

  	m.mb.RecordMysqlTableIoWaitTimeDataPoint(
  	now, s.timeFetch/picosecondsInNanoseconds, metadata.AttributeIoWaitsOperationsFetch, s.name, s.schema,
  	)

• For MysqlBufferPoolPages not sure if we even support uint64 and what options we have to send this value

Collector version

v0.110.0

[github-actions]

Pinging code owners:

• receiver/mysql: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

OTLP doesn't support uint64, unfortunately.

mysql.buffer_pool.pages is marked as non-monotonic sum. But such a huge number makes it sound like it's actually a plain monotonic sum. In that case, we can probably just drop the first bit of uint64, because we care more about the difference between the scrapes rather than the magnitude of each value...

[crobert-1]

Removing needs triage based on a path forward being proposed by a project maintainer.

[dloucasfx]

mysql.buffer_pool.pages is marked as non-monotonic sum. But such a huge number makes it sound like it's actually a plain monotonic sum. In that case, we can probably just drop the first bit of uint64, because we care more about the difference between the scrapes rather than the magnitude of each value...
@dmitryax
going through the logs, it does not look like it’s a monotonically increasing counter. I used awk to preserve the order.

perl -nle'print $& while m{failed to parse int64 for MysqlBufferPoolPages, value was\s+\K\d+}g' ~/Downloads/extract-2024-10-02T01_12_38.009Z.csv | awk '!a[$0]++'                         ok
18446744073709551264
18446744073709551265
18446744073709551266
18446744073709551271
18446744073709551267
18446744073709551276
18446744073709551272
18446744073709551270
18446744073709551284
18446744073709551306
18446744073709551269
18446744073709551275
18446744073709551285
18446744073709551287
18446744073709551321

Example of full log statement showing the timestamp and value not always incrementing:

1- 18446744073709551264 at 2024-10-02T01:12:07.891Z

"2024-10-02T01:12:07.891Z","""splunk-otel-collector-k8s-cluster-receiver-6864bbf5-pc6px""","""splunk-otel-collector""","2024-10-02T01:12:06.163Z        error   scraperhelper/scrapercontroller.go:197  Error         scraping metrics  {""kind"": ""receiver"", ""name"": ""mysql/primary"", ""data_type"": ""metrics"", ""error"": ""failed to parse int64 for MysqlBufferPoolPages, value was 18446744073709551264: strconv.ParseInt:    parsing \""18446744073709551264\"": value out of range"", ""scraper"": ""mysql""}"

2- 18446744073709551271 at 2024-10-02T00:21:24.036Z

"2024-10-02T00:21:24.036Z","""splunk-otel-collector-k8s-cluster-receiver-6864bbf5-pc6px""","""splunk-otel-collector""","2024-10-02T00:12:06.162Z        error   scraperhelper/scrapercontroller.go:197  Error         scraping metrics  {""kind"": ""receiver"", ""name"": ""mysql/primary"", ""data_type"": ""metrics"", ""error"": ""failed to parse int64 for MysqlBufferPoolPages, value was 18446744073709551271: strconv.ParseInt:    parsing \""18446744073709551271\"": value out of range"", ""scraper"": ""mysql""}

3- 18446744073709551269 at 2024-10-01T22:15:54.667Z

"2024-10-01T22:15:54.667Z","""splunk-otel-collector-k8s-cluster-receiver-6864bbf5-pc6px""","""splunk-otel-collector""","2024-10-01T22:06:06.155Z        error   scraperhelper/scrapercontroller.go:197  Error         scraping metrics  {""kind"": ""receiver"", ""name"": ""mysql/primary"", ""data_type"": ""metrics"", ""error"": ""failed to parse int64 for MysqlBufferPoolPages, value was 18446744073709551269: strconv.ParseInt:    parsing \""18446744073709551269\"": value out of range"", ""scraper"": ""mysql""}"