txhashset validation failing on testnet3 branch

[antiochp]

Fast-sync appears to be failing currently on testnet3.

We see a StoreErr(NotFoundErr, "wrapped") in the logs so I suspect something got clobbered with all the PR merges going on.

Jun 28 21:03:47.811 DEBG Header head before txhashset request: 83 / 4bbcf88a
Jun 28 21:03:47.811 DEBG Asking 127.0.0.1:13414 for txhashset archive at 64 8a3379a6.
Jun 28 21:03:47.880 DEBG handle_payload: txhashset archive for 8a3379a6 at 64 rewind to 152/138
Jun 28 21:03:47.903 DEBG leaf_set: copying rewound file .grin/txhashset/output/pmmr_leaf.bin.8a3379a6 to .grin/txhashset/output/pmmr_leaf.bin
Jun 28 21:03:47.904 DEBG leaf_set: copying rewound file .grin/txhashset/rangeproof/pmmr_leaf.bin.8a3379a6 to .grin/txhashset/rangeproof/pmmr_leaf.bin
Jun 28 21:03:47.910 DEBG txhashset: validated the output|rproof|kernel mmrs, took 0s
Jun 28 21:03:48.050 DEBG txhashset: verified 71 kernel signatures, pmmr size 138, took 0s
Jun 28 21:03:52.312 DEBG txhashset: verified 33 rangeproofs, pmmr size 152, took 4s
Jun 28 21:03:52.313 DEBG Error returned, discarding txhashset extension: StoreErr(NotFoundErr, "wrapped")
Jun 28 21:03:52.314 ERRO Failed to save txhashset archive: StoreErr(NotFoundErr, "wrapped")
Jun 28 21:03:52.314 DEBG handle_payload: txhashset archive for 8a3379a6 at 64, DONE

Have not had time to investigate but it appears to be happening somewhere in the txhashset validation step.

[yeastplume]

After a round of error handling improvements via #1208, the output is now:

Jun 29 15:47:21.648 DEBG txhashset: verified 222 rangeproofs, pmmr size 438, took 17s
Jun 29 15:47:21.872 DEBG Error returned, discarding txhashset extension: Store Error: NotFoundErr("BLOCK HEADER: ffffffff")                        
 Cause: DB Not Found Error: BLOCK HEADER: ffffffff           
 Backtrace: stack backtrace:                                                                                            
   0:     0x55ef75db49b5 - backtrace::backtrace::libunwind::trace::h42b75bce61037307
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.8/src/backtrace/libunwind.rs:53
                         - backtrace::backtrace::trace::ha6a62883bec4c260
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.8/src/backtrace/mod.rs:42
   1:     0x55ef75db01ee - backtrace::capture::Backtrace::new_unresolved::h1d576ccc48d4a120
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.8/src/capture.rs:88
   2:     0x55ef75daed67 - failure::backtrace::internal::InternalBacktrace::new::hb4dea2d348efe354
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.1/src/backtrace/internal.rs:44
   3:     0x55ef75dad82c - failure::backtrace::Backtrace::new::hda183b83df5c2f0f
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.1/src/backtrace/mod.rs:98
   4:     0x55ef7590b50d - <failure::error::Error as core::convert::From<F>>::from::h6c4026a0b0b58521
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.1/src/error.rs:34      
   5:     0x55ef758ffea3 - <T as core::convert::Into<U>>::into::hf40876e865866af2
                        at /checkout/src/libcore/convert.rs:396           
   6:     0x55ef7592877b - <failure::context::Context<D>>::with_err::hb90a2ada83e58899
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.1/src/context.rs:78
   7:     0x55ef758fff61 - failure::Fail::context::h505e20e7d8ae8d54
                        at /home/mcordner/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.1/src/lib.rs:127
   8:     0x55ef75931f43 - <grin_chain::error::Error as core::convert::From<grin_store::lmdb::Error>>::from::hf8fc5c6f9703106a
                        at chain/src/error.rs:202                     
   9:     0x55ef7596df09 - grin_chain::txhashset::input_pos_to_rewind::h4290faa4dc73363a
                        at chain/src/txhashset.rs:1136                                
  10:     0x55ef759684e8 - grin_chain::txhashset::Extension::rewind::h515ff048727460bd
                        at chain/src/txhashset.rs:780                                
  11:     0x55ef7596cfb0 - grin_chain::txhashset::Extension::verify_kernel_history::h7bd5e34a8793123b
                        at chain/src/txhashset.rs:1061                                                                                                                                                                                                                                                                        
  12:     0x55ef7596a9e9 - grin_chain::txhashset::Extension::validate::hea46950b68e12eeb
                        at chain/src/txhashset.rs:932                              
  13:     0x55ef759072ae - grin_chain::chain::Chain::txhashset_write::{{closure}}::hd7a011f1521afcad
                        at chain/src/chain.rs:551                                              
  14:     0x55ef758f7236 - grin_chain::txhashset::extending::he7b6f5dac3960e78
                        at chain/src/txhashset.rs:330
  15:     0x55ef75915ce8 - grin_chain::chain::Chain::txhashset_write::h6a01414f6de2af25
                        at chain/src/chain.rs:547
  16:     0x55ef7526ca36 - <grin_servers::common::adapters::NetToChainAdapter as grin_p2p::types::ChainAdapter>::txhashset_write::h3789fec5db671d3a

[antiochp]

Fixed in #1218.