Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR avoids writing a
_tmp_raw_data
array in calls to_recreate_raw_data
, preventing an allocation that in the past could consume lots of memory, causingdelete_versions
to fail. Closes #273.NOTE: This is a WIP PR - right now I'm seeing a really weird issue with the tests in
test_replay.py
where calls tonp.testing.assert_equal
are failing on tests which have had deleted versions. I'm guessing this is some issue flushing to disk; creating this PR here for further investigation.Okay, confirmed that closing and opening the file objects in the offending tests seems to resolve the issue. Will look for a way to flush output to disk to close this out.
The intermittent test failures are indeed resolved by calling
gc.collect
at the end ofdelete_versions
. Clearly this isn't ideal, but at the moment this seems like a functioning approach at least.Changes
_recreate_raw_data
now directly writes the recreated raw data to the file. Before, we were writing to a numpy array, then an intermediate_tmp_raw_data
dataset, then we'd move the_tmp_raw_data
toraw_data
to replace the original raw data. This should resolve any issues related to memory usage, and will likely also speed updelete_versions
overall, although we now pay an additional garbage collection penalty.posixpath
alias, which waspp
and thus would interfere with debuggerpp
pretty-print commands, in.