Improve behavior on corrupted checkpoint.

[fulmicoton]

If a checkpoint contains an invalid position (for instance not u64) for ingest,
we currently panic.

Ideally we should:

• log an error
• repair the checkpoint by removing the corrupted partition
• start indexing the faulty partition from the beginning.

This is very defensive and hence low priority

[jmintb]

Can I work on this? :)

[guilload]

Yes, you can.

Logging an error and skipping the partition is enough. I don't think self-clean-up is helpful because it's likely that the issue comes from either a bug in the source or a user manually editing a checkpoint. In the first case, the bug will reoccur, and self-cleanup will be less helpful. In the second, I'd rather have users clean up their mess themselves and decide from which point they want to resume indexing the partition. Always restarting from the beginning will yield duplicates.

[jmintb]

Perfect, do you know whereabouts in the codebase the panic(s) occurs? Sounds like it is in quickwit-ingest.

[guilload]

if you rg -e 'expect\(.*offset.*' in quickwit/quickwit-indexing/src/source you should find them all.

[jmintb]

Are there any existing tests or test data that would simulate this scenario?