remote write: reloading config with modified queue parameters results in duplicate metric registration

[beorn7]

What did you do?

Changed remote write queue_config fields like max_shards, capacity, max_samples_per_send.
SIGHUP'd the server.

What did you expect to see?

New config to be applied.

What did you see instead? Under which circumstances?

A panic.

After the following restart, the server came back OK. It's just that with a runtime config reload, the metrics of the new remote write queues collide with the old metrics.

Somehow, the unregister call added in #7188 isn't called in time or doesn't work as expected.

Environment

• Prometheus version:

  2.19.1

• Logs:

2020-06-24 15:02:25	
	/app/vendor/github.com/oklog/run/group.go:37 +0xbb
2020-06-24 15:02:25	
created by github.com/oklog/run.(*Group).Run
2020-06-24 15:02:25	
	/app/vendor/github.com/oklog/run/group.go:38 +0x27
2020-06-24 15:02:25	
github.com/oklog/run.(*Group).Run.func1(0xc000b001e0, 0xc000b00120, 0xc000912000)
2020-06-24 15:02:25	
	/app/cmd/prometheus/main.go:607 +0x1ec
2020-06-24 15:02:25	
main.main.func14(0x0, 0x0)
2020-06-24 15:02:25	
	/app/cmd/prometheus/main.go:817 +0x262
2020-06-24 15:02:25	
main.reloadConfig(0x7fffeda61ddc, 0x1e, 0x2f3b7e0, 0xc000c72cc0, 0xc000620340, 0x8, 0x8, 0x0, 0x0)
2020-06-24 15:02:25	
	/app/storage/remote/storage.go:77 +0xa6
2020-06-24 15:02:25	
github.com/prometheus/prometheus/storage/remote.(*Storage).ApplyConfig(0xc000188100, 0xc3d026fc00, 0x0, 0x0)
2020-06-24 15:02:25	
	/app/storage/remote/write.go:141 +0x3eb
2020-06-24 15:02:25	
github.com/prometheus/prometheus/storage/remote.(*WriteStorage).ApplyConfig(0xc000bcaa00, 0xc3d026fc00, 0x0, 0x0)
2020-06-24 15:02:25	
	/app/storage/remote/queue_manager.go:181 +0x10f2
2020-06-24 15:02:25	
github.com/prometheus/prometheus/storage/remote.newQueueManagerMetrics(0x2f76760, 0xc00019e730, 0xc755015ec0, 0x13, 0xc9885a4100, 0x40, 0xc771a42301)
2020-06-24 15:02:25	
	/app/vendor/github.com/prometheus/client_golang/prometheus/registry.go:400 +0xad
2020-06-24 15:02:25	
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc00019e730, 0xc0921c6000, 0xe, 0xe)
2020-06-24 15:02:25	
goroutine 75 [running]:
2020-06-24 15:02:25	

2020-06-24 15:02:25	
panic: duplicate metrics collector registration attempted
2020-06-24 15:02:25	
level=info ts=2020-06-24T13:02:25.313Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml

[csmarchbanks]

@beorn7 I think I know what the issue is, but to make sure, did this only happen when you changed a remote write config that has a name? Or does it happen on queues with auto-generated names as well?

[beorn7]

The reported case happened with named queues. We currently do not have queues with auto-generated names. (I guess if the auto-generated name changes because of the parameter changes, then there won't be a problem because the name is part of the registration.)

[codesome]

Fixed in #7452