Fix/2970

[jcnelson]

This fixes #2970 by ensuring that the unconfirmed state MARF trie is always committed when processing unconfirmed microblocks, even if the microblocks are invalid.

Thanks to @wileyj, I was able to identify the root cause of #2970: the node was processing an invalid microblock and then leaving the MARF open, which would then cause the node to crash when it attempted to mine a microblock (via InProgressError). The solution is to unconditionally close the unconfirmed state MARF trie, even if the unconfirmed microblock stream is bad. I have added a unit test that fails in develop but succeeds in this PR to demonstrate this.

Almost as concerning is how the invalid microblock came to be. The microblock in question was invalid because it contained a transaction with a bad nonce. Moreover, this node mined it -- it produced a microblock that was valid at the time it was created, but later became invalid through some unknown means. I need to figure out why this is, but for the time being, this PR addresses the crash.

[codecov]

Codecov Report

Merging #3016 (3b2f218) into develop (a289504) will increase coverage by 54.40%.
The diff coverage is 99.62%.

@@             Coverage Diff              @@
##           develop    #3016       +/-   ##
============================================
+ Coverage    28.15%   82.56%   +54.40%     
============================================
  Files          242      242               
  Lines       195403   195667      +264     
============================================
+ Hits         55025   161556   +106531     
+ Misses      140378    34111   -106267     

Impacted Files	Coverage Δ
src/chainstate/stacks/db/unconfirmed.rs	93.84% <99.62%> (+59.85%)	⬆️
src/chainstate/stacks/miner.rs	93.89% <100.00%> (+83.74%)	⬆️
src/main.rs	0.11% <0.00%> (+0.11%)	⬆️
testnet/stacks-node/src/run_loop/neon.rs	81.21% <0.00%> (+0.56%)	⬆️
src/cost_estimates/fee_medians.rs	93.27% <0.00%> (+0.89%)	⬆️
src/types/chainstate.rs	90.14% <0.00%> (+1.40%)	⬆️
src/cost_estimates/fee_scalar.rs	90.78% <0.00%> (+1.41%)	⬆️
src/burnchains/bitcoin/mod.rs	40.57% <0.00%> (+1.44%)	⬆️
src/vm/database/key_value_wrapper.rs	96.48% <0.00%> (+1.59%)	⬆️
src/chainstate/coordinator/comm.rs	85.98% <0.00%> (+2.80%)	⬆️
... and 200 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a289504...3b2f218. Read the comment docs.

[pavitthrap]

lgtm

[CLAassistant]

All committers have signed the CLA.

[jcnelson]

@pavitthrap can you sign the CLA? Thanks!

[pavitthrap]

@jcnelson could you add a note about this in CHANGELOG.md?