Increase LivenessCheck timeout

[rajagopalanand]

What this PR does:

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[rajagopalanand]

histogram_quantile(0.99, sum(rate(cortex_ruler_client_request_duration_seconds_bucket{operation="/ruler.Ruler/LivenessCheck"}[2m])) by (le,instance))

Depending on the environment, the LivenessCheck could take longer than the original 100 ms like the graph above shows. I want to set the timeout to 1s because it is not configurable

[yeya24]

Let's make it configurable then?
There might be cases where 1s timeout couldn't work

[rajagopalanand]

Let's make it configurable then? There might be cases where 1s timeout couldn't work

Wondering what would be the best way to do this. There is already a Remote timeout (remote_timeout) under ruler client which is set to 2 minutes. If I introduce a config specifically for LivenessCheck, I was thinking it should also go under ruler client but then it might be confusing to have a generic deadline (remote_timeout) and a specific one for LivenessCheck