Add logging during WAL replay

[csmarchbanks]

Add logging so users know what is happening while their Prometheus takes a long time to start up. For example, hopefully questions like this would be avoided, or focused on the correct issue: https://groups.google.com/d/msgid/prometheus-users/4e309664-50b5-46dd-bac3-5d3499928e92%40googlegroups.com

The new startup logs from Prometheus look like:

level=info ts=2019-07-13T17:14:46.167Z caller=main.go:652 msg="Starting TSDB ..."
level=info ts=2019-07-13T17:14:46.167Z caller=web.go:448 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-07-13T17:14:46.167Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1562160608909 maxt=1562162400000 ulid=01DFP576C2AAJ9819V4S1N191W
level=info ts=2019-07-13T17:14:46.167Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1562162400000 maxt=1562169600000 ulid=01DFP576DBV98QPVZ4RR09DSXW
level=info ts=2019-07-13T17:14:46.171Z caller=head.go:505 component=tsdb msg="replaying WAL, this may take awhile"
level=info ts=2019-07-13T17:14:46.172Z caller=head.go:529 component=tsdb msg="WAL checkpoint loaded"
level=info ts=2019-07-13T17:14:46.172Z caller=head.go:553 component=tsdb msg="WAL segment loaded" segment=122 maxSegment=126
level=info ts=2019-07-13T17:14:46.173Z caller=head.go:553 component=tsdb msg="WAL segment loaded" segment=123 maxSegment=126
level=info ts=2019-07-13T17:14:46.173Z caller=head.go:553 component=tsdb msg="WAL segment loaded" segment=124 maxSegment=126
level=info ts=2019-07-13T17:14:46.173Z caller=head.go:553 component=tsdb msg="WAL segment loaded" segment=125 maxSegment=126
level=info ts=2019-07-13T17:14:46.173Z caller=head.go:553 component=tsdb msg="WAL segment loaded" segment=126 maxSegment=126
level=info ts=2019-07-13T17:14:46.174Z caller=main.go:667 fs_type=EXT4_SUPER_MAGIC
level=info ts=2019-07-13T17:14:46.174Z caller=main.go:668 msg="TSDB started"

[brian-brazil]

👍

I was thinking just the same.

[krasi-georgiev]

LGTM

[cstyan]

👍 yes please, we already do this in the WAL watcher replay