BUG: Zip file output sometimes corrupted on Linux

[tbirdso]

Overview

Tests for reading from .zip file store sometimes fail. Exact cause is unclear at the moment.

First CI run (failed)
https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF/actions/runs/5716549674/job/15488329023?pr=35

Second CI run, no changes (passing):
https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF/actions/runs/5716549674/job/15490659684?pr=35

Workaround

Re-running CI seems to work for now.

[tbirdso]

This is currently blocking the v0.1.3 PyPI release: https://github.com/InsightSoftwareConsortium/ITKIOOMEZarrNGFF/actions/runs/5739761493

[tbirdso]

Following investigation in #44, it was observed that the memory.zip output of the inMemory_zip test is sometimes a corrupted archive. The exact cause for this is not clear. Behavior is observed regularly in CI and infrequently in local testing on Linux platforms.

The Tensorstore zip driver is maintained in the https://github.com/InsightSoftwareConsortium/tensorstore fork and is not maintained by Google. We will need to track this ourselves.

Following offline discussion, we will disable the failing test for now to prevent blocking PyPI releases and continue investigation in parallel with subsequent work.

[dzenanz]

After all the contents are added to in-memory zip file, the buffer size is 47447. After closeZip() is called, the buffer size increases to the correct value of 47685. This leads me to suspect that destructor is not invoked sometimes.

I will now try to fix this.

I originally wrote this in the wrong issue's discussion: #48 (comment)

[dzenanz]

Looking at tensorstore's commit log, there were some fixes recently regarding caches and deep copies. I will rebase our changes on top of current master, to see whether that resolves the problem.

[dzenanz]

With rebasing, I ran into a snag: google/tensorstore#110.

[dzenanz]

With building cleared up, the rebase does not help with crashes. We should still use the newer version of tensorstore, so I created this PR:
InsightSoftwareConsortium/tensorstore#1
Please just review, do not click merge - this branch should become the new master.

[dzenanz]

#54 fixes the problem via a workaround. I still think that the original problem was caused by the ~ZipEncapsulator() destructor not always being called. But digging through tensorstore internals seemed daunting.