rosetta-cli check with --lookup-balance-block=false doesn't find reconciliation errors unless --interesting-accounts is given

[metajack]

Describe the bug

In testing of a Rosetta implementation on a blockchain that should contain reconciliation errors, the check command fails to find the errors unless --interesting-accounts is used. This seems to be caused by the reconciliation checks becoming overly delayed so as to never happen.

To Reproduce

Run rosetta-cli check --lookup-balance-by-block=false --interesting-accounts interesting.json and watch it find reconciliation errors.

Run rosetta-cli check --lookup-balance-by-block=false and let it spin forever and never find any reconciliation problems.

Expected behavior

I expect the same errors to be found, though perhaps in different amounts of time.

[patrick-ogrady]

Thanks for posting this issue @metajack! You found a very annoying bug that only affects check when --lookup-balance-by-block=false.

This issue can occur if an account (in this case, ones that have reconciliation issues) are skipped while check is catching up to tip and never show up in blocks again.

When accounts are skipped, they are not added to the inactive reconciliation queue (either initially or when taken from the inactive queue 😬 ): https://github.com/coinbase/rosetta-sdk-go/blob/6fcf086bc01df22380c3809332641cb870683d9b/reconciler/reconciler.go#L446-L456

This behavior also occurs if we are less than the highWaterMark: https://github.com/coinbase/rosetta-sdk-go/blob/6fcf086bc01df22380c3809332641cb870683d9b/reconciler/reconciler.go#L275-L277

We will work a fix in for this very soon!