Make OSD Bootstrap more sturdy in CI

[Swiddis]

Description

As the Bootstrap command in CI is inconsistent, this has been causing a lot of delays in getting CI feedback in PRs. This PR seeks to address this with a few safeguards:

1. Advice from the OSD Developer Guide was applied to update the network timeout. In their own CI they use this network setting and also have a backup call in case it fails. They also use a clean which may address some inconsistency, though it should be a no-op in a CI setting.
2. fail-fast is disabled in the CI, so even if the measures from (1) are unsuccessful, we will at least receive some feedback from the other two runs without needing a full re-run.
3. The results are cached to both reduce the effects of the issue and speed up CI overall.

Issues Resolved

N/A

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #901 (ea657f3) into main (43e921e) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #901   +/-   ##
=======================================
  Coverage   43.72%   43.72%           
=======================================
  Files         313      313           
  Lines       18629    18629           
  Branches     4534     4534           
=======================================
  Hits         8146     8146           
  Misses       9932     9932           
  Partials      551      551           

Flag	Coverage Δ
dashboards-observability	43.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[pjfitzgibbons]

@Swiddis Love this!
Do you have any way to test this in local dev? (docker instructions somewhere?)
If so, can you post a terminal log of the test. I just want to get a feel for what this actually does at CI runtime?

[Swiddis]

Do you have any way to test this in local dev? (docker instructions somewhere?)

Not at the moment, I'm mostly just relying on an actual run to hang. It looks like this one is pretty close to expected behavior, since it hung for multiple minutes while waiting on network calls. I'm still experimenting a bit with caching.

[Swiddis]

Results are in

OS	No Cache	Cache
Ubuntu	268s	20s
Windows	711s	124s
Mac	534s	132s

Should be good to merge now

[pjfitzgibbons]

Is this seconds?

[Swiddis]

Milliseconds, this PR uses the same number as is used in the OpenSearch Dashboards CI, which in turn is inspired by a related issue on yarn that suggests a similar approach with a number that's also in that ballpark.

[joshuali925]

why is it caching target?

[Swiddis]

It showed up when I scanned for what files bootstrap was making, I assumed it may be important. I don't know the technical details of what bootstrap does, so I might have overselected.

[joshuali925]

i'm not sure why bootstrap would touch target. target is the transpiled javascript from source code (public ts/tsx) to be executed in browser, yarn start (optimizing phase) and yarn build would update target.

caching it probably won't cause an issue even if your key does not consider source code changes (because yarn build will update target anyways), but i don't think it should be cached. it's not really related to dependencies

[joshuali925]

this is ok but there's also a retry workflow https://github.com/marketplace/actions/retry-step