List of spec tests not covered by Lodestar

[q9f]

I compared the tag release v0.30.0 (alpha) which ran pipeline 3619666497 with the upstream spec test at tag v1.1.0-beta.3. I have put my findings in the linked Gists (if any).

Phase 0

• general/phase0/bls/ 60/60 (100%) ✅
• general/phase0/ssz_generic/ 1795/1905 (94.2%, 110 not covered)
• mainnet/phase0/ 712/712 (100%) ✅
• minimal/phase0/ 3912/3919 (99.8%, 7 pending)

I don't know what pending means in CI, they are implemented but not run? Or timeout?

Altair

• general/altair/bls/ 0/20 (0.0%, 20 not covered)
• mainnet/altair/ 459/504 (90.7%, 45 not covered)
• minimal/altair/ 3675/4784 (76.8%, diff too large/unknown)

It's not an easy process to create these diffs, so I did not look into the last large diff due to time constraints. Phase 0 and Mainnet should be prioritized though. Also the merge diffs are huge (obviously), so I didn't bother to list them.

[q9f]

It would be beneficial to quickly get a list of available tests in the consensus repository. So I used a ruby script to parse the contents and generate the lists.

Also, I couldn't use the TypeScript utilities that are supposed to check test coverage. So, instead, I parsed the CI logs to get a clear image of what's being run. Also, here, it would be nice if I could figure out a way in the future, to access the list of covered tests more quickly in the future.

[dapplion]

Thank you @q9f for being on top of these issues ❤️

Forgot this issue existed, now it's linked with the relevant PR