This repository has been archived by the owner on Aug 13, 2019. It is now read-only.

prometheus_tsdb_compactions_failed_total metric not incremented when compaction fails #612

Closed

vsliouniaev opened this issue May 27, 2019 · 1 comment · Fixed by #613

Labels

bug help wanted low hanging fruit

vsliouniaev commented May 27, 2019

We have had a corrupted WAL after a disk issue and got the following error messages in the logs

May 20 01:01:59 q2-gog-sh00-pm0 prometheus[3157]: level=error ts=2019-05-20T01:01:59.300Z caller=db.go:363 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: create checkpoint: read segments: corruption in segment /datadrive/prometheus/db/platform/wal/00225282 at 32768: unexpected first record, dropping buffer"

however, the value of prometheus_tsdb_compactions_failed_total isn't increasing when this happens.

The text was updated successfully, but these errors were encountered:

krasi-georgiev added bug help wanted low hanging fruit labels

Contributor

krasi-georgiev commented May 27, 2019

thanks for the report, this is indeed a bug and it should be easy to fix just by moving the logic for the metric into a different func.

krasi-georgiev mentioned this issue

fix the "failed compaction" metric. #613

Merged

krasi-georgiev closed this as completed in #613

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.