Memory optimizations during data verification

[PadmaB]

Improvement

Improve memory consumption during data verification

Definition of Done

• Knowledge is distributed: Have you spread your knowledge in pair programming/code review?
• Unit tests are provided: Have you written automated unit tests?
• Integration tests are provided: Have you written automated integration tests?
• Minimum API exposure: If you have added/changed public API, was it really necessary/is it minimal?
• Operations guide: Have you updated the operations guide about ops-relevant changes?
• User documentation: Have you updated the READMEs/docs/how-tos about user-relevant changes?

[shreyas-s-rao]

The ReadAll() function exposed by the WAL package by etcd is the only way, at present, to read through the contents of the wal logs. The function returns the entire contents of the wal back to the caller in-memory, as an array. This makes the memory consumption proportionate to the wal size itself, and is undesirable.

Our use case (wal content verification) does not require us to fetch the contents of the wal, but to simply ensure that the read is successful, the wal opens at the given snapshot, and there are no CRC mismatches.

This calls for a new function Verify() with a streamlined functionality, that simply goes through the wal records, does CRC checks and snapshot mismatch checks, and returns just the error encountered. This would be a task on the upstream etcd repo, as we will have to raise an issue and maybe also a PR to address our use case.

[shreyas-s-rao]

I have raised an issue on the upstream etcd repo at etcd-io/etcd#10497 regarding the feature request for a wal.Verify() function.

[shreyas-s-rao]

I had raised a PR on upstream etcd (etcd-io/etcd#10516) and it has now been merged. Still waiting for an update on how and when the feature will be added to a release/patch. Hoping it will be part of v3.3.13 (next patch version).