Spec and Implement the end of life of a shard.

[fulmicoton]

We need to clean up data from the metastore, from the ingester and from the control plane.

The indexer reports in the metastore and in chitchat that it has reached EOF.

• On publish the indexer updates the shard position to ~eof. The list of shards is part of the control plane view but not their position: We do not want publish to require the control plane to be up.
• the indexer also publish its EOF status on chitchat, which is eventually caught up by the control loop control_running_plan routine.
• the control plan attempts to remove the data from the ingester. This call may fail. (best effort / optional)
• The control plane removes the data from the metastore
• The control plane removes the data from its internal model.

If the ingester is unreachable, there is no way to know whether it will ever come back or not.
We should therefore have a GC like process to tell an ingester the list of shards it is supposed to host, and the ingester should remove the diff.

If a shard never reaches the status of EOF because all of its replicas have been disconnected, then we also need a TTL based solution to remove it. After this removal, the data is lost forever.

• creation of shard positions service that create an evntually consistent view of the published position of shards, connected to the event broker.
• hooking the indexing service to the shard positions service
• hooking the ingesters to the shard positions, so that they can truncate their queues when they receive an update.
• hooking the control plane to delete shards from the metastore when EOF is detected
• have a code path to entirely delete queues from ingesters