Consistently have "CI Green" for Lotus - (Internal improvements program)

[BlocksOnAChain]

We plan to deliver this as a part of our "Internal improvements day" and improve our Lotus codebase/velocity.

As we discussed, the main actions which will enable us to achieve this state, are:

• Bolster code coverage program - all code that is touched forward (existing and new code) needs to have 70+% coverage so we bring overall coverage up with time
• Flaky test - eradication program - fix/solve flaky tests and build a CI model that will enable us to catch them early on
  • Dashboard: https://app.circleci.com/insights/github/filecoin-project/lotus/workflows/ci/tests?branch=master&reporting-window=last-30-days

[BigLep]

The "bolster code coverage program" requirements are higher than the base level I belive.

Base Level Requirements

The baselevel thing we want to avoid is someone being able to submit a change on a .go source file without any accompanying tests.  Even better is to make sure the code coverage level doesn't drop.

Implementation Plan

Per @Stebalien , we can't have a rule that a given change always have >= 0% code coverage change. There is some nondeterminism in the runs. We have to go with a threshold like ≥ -.1% to account for this. Furthermore, patch-based code-coverage fails because a patch that edits a likely impossible error message/panic will always have zero coverage.

We'd need to define separate "projects" in our codecov config for each lotus subsystem package (ish, use discretion). Each project would have:

1. The target set to auto. This will make codecov "hold" the current coverage.
2. A threshold of 0.1%. Looking at our tests, I see code coverage go up and down by about due to noise 0.07% so 0.1% shouldn't fail. If it becomes a problem, we can tune this value for "noisy" sub-projects.
3. informational set to true if we want coverage to pass regardless. This will most likely be useful for cli and maybe api, where a lot of code is boilerplate or CLI commands that won't be covered by unit tests. NOTE: we often just ignore these directories, but informational coverage is likely better.

[BigLep]

It would be ideal to say that all new code had 70+% coverage, but there are going to be exceptions to this (e.g., 1. It's hard to ask contributors to write regression tests for one-line fixes in untested code. 2. Sometimes testing a case is non-trivial. Ideally, we'd go and write the test anyways. But when the bug is a really low priority (e.g., random community patch, random panic in some nearly impossible edge-case, etc.), this can be hard to prioritize.). Given the focus right now is to establish the mindset that "we only merge if the CI is green", we're biasing towards not enforcing new code must have higher coverage so as to avoid exceptions. Code reviewers will have the code coverage report data and can request coverage as needed. This is admittedly a best intention, but we're trying to establish some reliable foundation before we ratchet up.

[Stebalien]

Ok, so most of CI is green now. Except that I re-enabled the testrgound tests (because the infrastructure was fixed) and now we're seeing a test failure: #7186.

[BlocksOnAChain]

closing since we have been in the "green" status for some time now.