Assert in status_cache::save_failure_status

[pgarg66]

Problem

A node is crashing in testnet with the following stack trace. This is happening on v0.12 branch. It might be reproducible on master branch as well. Current tip of v0.12 is at bdee3a2 commit.

[2019-03-22T17:17:27.723971136Z WARN  solana_runtime::bank] =========== FIXME: commit_transactions() working on a frozen bank! ================
thread 'solana-banking-stage-tx' panicked at 'sig not found', runtime/src/status_cache.rs:103:9
[2019-03-22T17:17:27.723986752Z WARN  solana_runtime::bank] =========== FIXME: commit_transactions() working on a frozen bank! ================
[2019-03-22T17:17:27.725580583Z WARN  solana::cluster_info] DHEkgopESGeHsD32TpU72TG9MyAanZHkEBeGy6TpzNbc: LEADER_UPDATE TO DHEkgopESGeHsD32TpU72TG9MyAanZHkEBeGy6TpzNbc from DHEkgopESGeHsD32TpU72T
G9MyAanZHkEBeGy6TpzNbc
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:70
   2: std::panicking::default_hook::{{closure}}
             at src/libstd/sys_common/backtrace.rs:58
             at src/libstd/panicking.rs:200
   3: std::panicking::default_hook
             at src/libstd/panicking.rs:215
   4: solana_metrics::metrics::set_panic_hook::{{closure}}::{{closure}}
   5: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:482
   6: std::panicking::begin_panic
   7: <solana_runtime::status_cache::StatusCache<T>>::save_failure_status
   8: solana_runtime::bank::Bank::commit_transactions
   9: solana::banking_stage::BankingStage::process_and_record_transactions
  10: solana::banking_stage::BankingStage::process_packets
  11: solana::banking_stage::BankingStage::process_loop

Proposed Solution

Triage and fix.

[mvines]

@pgarg66 - assigning to you to check if this still reproduces on v0.13, if it then makes sense to move this issue to somebody else go for it

[pgarg66]

👍

[pgarg66]

I haven't noticed this crash lately. Closing the issue.