Investigate Consistent CI Failure for Follower Bootup Integration Test

[AshtonStephens]

Investigate Consistent CI Failure for Follower Bootup Integration Test

A number of PRs began failing integration tests after seemingly no changes were made to the areas of code that would impact the integration tests. The purpose of this ticket is to identify why this one test uniformly fails and define a path forward to either resolving or ignoring it.

• Example of the failure: https://github.com/stacks-network/stacks-core/actions/runs/8526499949/job/23356470025?pr=4583

This is blocking the following PRs submitted by @jferrant:

• Consider nonce when checking for existing vote transactions in stacke… #4621
• Do not attempt to treat miner as a coordinator when updating DKG #4591
• Feat/stacks signer fee estimate #4583 (though there are other failures on this one, so there's likely a separate issue here as well)

[obycode]

This test passes locally on next.

[obycode]

Passes locally on bugfix/resubmit-dkg-vote-inc-nonce (#4621) as well.

[obycode]

Passes locally on bugfix/enable-dkg-in-current-reward-cycle (#4591).

[obycode]

Passes locally on feat/stacks-signer-fee-estimate (#4583) as well.

[obycode]

The issue just seems to be that it takes too long and times out in CI.

[obycode]

It takes about 4 minutes to run locally, but is timing out after 30 minutes in the CI runner. I'm not sure what the best approach should be for tests like this. It might be nice to ignore it during CI, but have a daily job that runs all the slow tests. @wileyj do we have anything like that for other slow tests at the moment?

[AshtonStephens]

@obycode Thank you so much for the quick research. Truly an MVP

[wileyj]

It takes about 4 minutes to run locally, but is timing out after 30 minutes in the CI runner. I'm not sure what the best approach should be for tests like this. It might be nice to ignore it during CI, but have a daily job that runs all the slow tests. @wileyj do we have anything like that for other slow tests at the moment?

we do - it's a workflow literally called "slow-tests". was literally speaking to @jferrant in slack about this.

https://github.com/stacks-network/stacks-core/blob/master/.github/workflows/slow-tests.yml#L33C1-L34C63

this worfklow has 2 tests that i identified a while back as tests that can end up running indefinitely, but if they are to pass they each take around 20 minutes on a github runner. these tests are only run during a release process, but can be run manually as a github action with the https://github.com/stacks-network/stacks-core/actions/workflows/standalone-tests.yml workflow (select "slow tests" from the dropdown).

running these daily won't give us anything useful - the tests in that slow-tests workflow will either work or they won't and because it's a black blox they're running on, it's rare we'll know why.
but what i've seen is that the test is simply waiting for something to happen that never will.

i recommended to @jferrant that this specific test, if it's going to be unreliable, is to move it to the slow-tests workflow.

[wileyj]

i also have seen this test pass or fail on the same commit, so i have a strong feeling it's what i wrote above - the workflow is just waiting for the test to do something that is either taking too long or will never happen for reasons.
we can increase the timeout, but if the test passes, it's pretty quick - i see it complete in < 10 minutes reliably when it passes.