Add logic to prune all historic states

[michaelsproul]

Description

It would be useful for several purposes to be able to drop all of the states in the freezer DB. Some uses include:

• Recover from database corruption (e.g. due to Fix bug in freezer DB storage of randao_mixes #3011).
• Enable historic block pruning (see Historic block pruning #4158).
• Save space by running without historic states (Add a mode to run without freezer states #3211).
• Migrate from stable Lighthouse to tree-states

Steps to resolve

I think for forwards-compatibility with tree-states it would be best to implement the deletion as several iterators over the relevant columns, which just delete everything in the column without parsing it. Basically:

for column in relevant {
  for key in db.iter_column(column) {
    to_delete.push(key);
  }
}

Where relevant consists of:

• DBColumn::BeaconStateSummary (where the ColdStateSummary is stored)
• DBColumn::BeaconState (where the PartialBeaconState is stored)
• DBColumn::BeaconRestorePoint (where the restore point -> state root index is stored)
• DBColumn::BeaconStateRoots
• DBColumn::BeaconHistoricalRoots
• DBColumn::BeaconRandaoMixes
• DBColumn::BeaconHistoricalSummaries

I think on unstable the beacon block roots should be left intact, as we expect them to exist whenever we've stored historic blocks in the DB.

Atomically with those deletions we should update the state_upper_limit in the Anchor to reflect the fact that those states are gone. I think setting it to u64::MAX could be a good way to signal a new mode of operation where "this node isn't storing any historic states". Alternatively, we set it to the next restore point slot (or snapshot slot for tree-states) which is >= split.slot. Either way, we need this block of logic from the tree-states branch that prevents the finalization migration from trying to write bad states:

lighthouse/beacon_node/store/src/hot_cold_store.rs

Lines 2204 to 2211 in 079cd67

	// Do not try to store states if the first snapshot is yet to be stored.
	if anchor_info
	.as_ref()
	.map_or(false, |anchor| slot < anchor.state_upper_limit)
	{
	debug!(store.log, "Skipping cold state storage"; "slot" => slot);
	continue;
	}

I think we probably also want to solve the alignment issues for checkpoint sync first, otherwise we'll have to avoid deleting all the states when the split point is at a skipped slot. See: #3210 (comment).

[michaelsproul]

I'm going to work on this now that #4610 is merged. We should be able to get a lighthouse db prune-states command working with relative ease 🤞