[Heartbeat] improve monitor state loader failure attempts

[vigneshshanmugam]

Description

PR addresses couple of things on the state loader

• We were waiting unnecessary for more than 4 seconds after all of the loading attempts.
• Failure attempts were too frequent like the connection was reestablished in 0, 135 ms, 1200ms, 4300 ms, Now with the change, we will start from 0, 1200ms, 4300ms and exit with error code.
• Improve the logs around the last attempt and keep the message consistent.

How to test this PR locally

• Create a heartbeat browser/lightweight monitor
• Spin up a local ES
• Close the local ES once the initial connection gets established
• Wait for the monitor to get finished, then check the attempts.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @vigneshshanmugam? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[dliappis]

Hey @vigneshshanmugam ,

Is this ready for review? I can confirm that the failures bubbled up by CI are reproducible and don't exist e.g. on the regular runs on main https://buildkite.com/elastic/heartbeat/builds/5135
I was also able to also reproduce the hanging (Go) unit tests:

=== Failed
=== FAIL: heartbeat/monitors TestMonitorBasic (1.00s)
    monitor_test.go:103: No publishes detected!

=== FAIL: heartbeat/monitors TestMonitorCfgError (1.00s)
    monitor_test.go:103: No publishes detected!

=== FAIL: heartbeat/monitors/active/http TestUpStatuses (unknown)


=== FAIL: heartbeat/monitors/active/http TestUpStatuses/Test_OK_HTTP_status_208_using_hosts_config_field (unknown)
signal: interrupt

which I suppose is the reason why you tried adding an explicit timeout in the Buildkite steps.

[vigneshshanmugam]

@dliappis Thanks for the link on the main branch, there was a bug in the code that caused the timeouts and made me think increasing would help. Should be fixed in the next run 👍🏽

[emilioalvap]

LGTM

[dliappis]

LGTM (all CI is green, no changes related to ingest-eng-prod owned codebase)