[Remote Store] Fix replica failing to download segment files from remote store - NoSuchFileException

[linuxpi]

Describe the bug

• While replica is hydrating from remote store, primary can delete the segments from remote if there is merge and old segments files are marked for deletion.
• We need to ensure replica shard doesn't fail due to unable to fetch old segment files which have been marked stale by primary shard
• Today as a safeguard to this, we retain all segment files for last N metadata files. This doesn't guarantee we wont run into this issue but since the N is dynamically configurable, we have a mitigation if a cluster faces this issue.
• But this leads to unnecessary segment files retained in remote store which are stale, hence increasing the remote store usage and cost.
• With a proper solution in place, we can also remove the current logic to retain segment files corresponding to last N metadata files instead we can just keep the latest one.

** Possible Solutions **

• We can acquire locks on the segment files before replica starts hydrating from remote, to make sure the files are not deleted in remote. We already have a framework in place for such behavior which is used by shallow snapshots with remote store
• Another alternative might be to automatically restart recovery on replica if it fails with NoSuchFilesException. But this might not be deterministic and can also lead to false positives.