Lily tipset-worker skips tasks when node desyncs

[frrist]

Describe the bug:

When a tipset-work job is running on a lily node that desyncs from the chain and a task is run for chain state the lily node has not synced yet it will fail to process the task:

{"level":"error","ts":"2022-06-29T19:38:30.591Z","logger":"lily/distributed/worker","caller":"queue/worker.go:90","msg":"task failed","tipset":"{bafy2bzacecnefd7q6zhhzxeq7u5ehkwcbwfnnblfkpz6acr6lskfenlgmwzt2,bafy2bzacebhpftzsoiorblqkgciossg2epijkres5fwtrronri546pcdyjmbm,bafy2bzacedttx62ue2ezlsyggeymicfloiwsjp7h6diafppnqxrhqzqa2uwoq,bafy2bzacebly27ss4fzf2ezrojk4yuxdfczkr44lgrazmjivbrtwb5v4t2azq}","height":1940832,"tasks":"[block_header block_parent drand_block_entrie miner_sector_deal miner_sector_infos_v7 miner_sector_infos miner_sector_post miner_pre_commit_info miner_sector_event miner_current_deadline_info miner_fee_debt miner_locked_fund miner_info market_deal_proposal market_deal_state message block_message receipt message_gas_economy parsed_message internal_messages internal_parsed_messages multisig_transaction chain_power power_actor_claim chain_reward actor actor_state id_address derived_gas_outputs chain_economics chain_consensus multisig_approvals verified_registry_verifier verified_registry_verified_client]","type":"tipset:index","error":"indexing tipset {bafy2bzacecnefd7q6zhhzxeq7u5ehkwcbwfnnblfkpz6acr6lskfenlgmwzt2,bafy2bzacebhpftzsoiorblqkgciossg2epijkres5fwtrronri546pcdyjmbm,bafy2bzacedttx62ue2ezlsyggeymicfloiwsjp7h6diafppnqxrhqzqa2uwoq,bafy2bzacebly27ss4fzf2ezrojk4yuxdfczkr44lgrazmjivbrtwb5v4t2azq}.(1940832) taskID e0c8e27a-b8c4-4981-9ca2-ed5d9029e4c4: Error get circulating supply: load state tree: failed to load state tree bafy2bzaceavp4ie5pnlkezr5osczuv24zi6e2zehgdmpfdc3jlxhzcy3esimc: failed to load hamt node: blockstore: block not found : skip retry for the task"}

It then will cancel processing the task due to: https://github.com/filecoin-project/lily/blob/v0.10.1/chain/indexer/distributed/queue/tasks/index.go#L96

The tipset-worker should reschedule the task since the state will become available later, or because another node may already have the state and thus be capable of completing the task.

Lily Version: v0.10.1