Delete completed local blocks when replaying wal

[mdisibio]

What this PR does:
This PR fixes #937 by deleting locally complete blocks during wal replay, and another race condition on ingester startup. Here is a description of the situation and why this should fix it.

1. An ingester may restart or crash while completing the wal file. It can happen after creating the new local block, but before deleting the wal file.
2. On restart the ingester replays wal, and rediscovers local blocks. Both exist it ends up with 2 queue entries instead of 1. It will both recomplete the wal and reflush the local block. These occur independently and there are no consistency checks while processing the queue entry.
3. This is a race condition in multiple ways.
   a. The wal complete is in progress, writing over top of data when it is flushed to the backend. The flush will reach EOF and assume all went well. This creates a partially written block in the backend.
   b. The wal complete can entirely finish before the flush. It appends another entry into completedBlocks. When the flush occurs it reads the first entry from completedBlocks and saves it as the meta data. This is the actual condition for magic number errors and incorrect meta. It uses the meta as rediscovered in step 2, which may be a different encoding, etc.
   c. The wal complete can start on a new block before rediscoverLocalBlocks begins. rediscoverLocalBlocks sees a bad block (missing meta) and deletes it. I think this ends up in a situation similar to (a) but different causes. This is handled by not rediscovering the local block is there is still a wal for it.

A couple more notes:

• Enabling search increased the risk of these errors because it added more work between creating the local block and deleting the wal in step 1.
• We've also seen issues with partially flushed blocks like 3a and 3b, but afaik there isn't an issue tracking them.

Which issue(s) this PR fixes:
Fixes #937

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]