Stream reflectiontable msgpack directly to file #1115
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The previous implementation held the reflection table up to 3 times in memory during the output phase. This implementation passes the python filehandle into C++ space and then writes the msgpack output directly to the file. As a result memory consumption is significantly reduced. Fixes #1112
|
Fails on Python3 because boost_adaptbx streambuf is not Python3 compatible. Presumably this was never noticed because the tests have been intentionally broken for half a year |
|
Compared maximum memory use in kBytes on writing between pickle, existing msgpack implementation and msgpack streaming method in development.
|
|
Compared maximum memory use in kBytes on reading between pickle and the existing msgpack implementation. A streaming read implementation is possible but would very likely be a lot more complex than the current implementation.
|
Fix compilation Compiler appears confused between iotbx::mtz::object and boost::python::object. Be explicit that we want the latter.
Functions that encapsulate up to three separate functions and distinguish which one to call based on a string parameter are not sane. Refactor the test extension functions into simple, flat functions with a sane API. Remove unused or irrelevant code.
instead of a home-grown solution.
The streambuf interface deals exclusively with binary data. We need to be explicit about this so that it works in Python 3. So make functions return bytes, not strings. Incidentally, we do not need a precompiler macro, as in Python 2 PyBytes == PyString
Otherwise ostream only flushes on Python garbage collection, which may not happen at deterministic times.
|
Thanks to @dagewa, @ndevenish, and @graeme-winter for their assistance in getting this PR off the ground. |
|
DIALS and xia2 full regression tests passed (tested on Python 2 only) |
dagewa
approved these changes
Jan 30, 2020
Anthchirp
added a commit
that referenced
this pull request
Feb 3, 2020
The previous implementation held the reflection table up to 3 times in memory during the output phase. This implementation passes the python filehandle into C++ space and then writes the msgpack output directly to the file. As a result memory consumption is significantly reduced.
Merged
Anthchirp
added a commit
that referenced
this pull request
Feb 10, 2020
* Make resolution estimation more stable in presence of ice and powder rings and with small molecule data (#1097) * Fix spot finding and integration of files with index 0 (#1128, cctbx/dxtbx#133) * Fix cutoff value on recent data files from DLS I03 (cctbx/dxtbx#136) * Reduce memory usage when writing .refl files (#1115) * `dials.integrate`: Fix broken memory check in cases of high multiplicity (#1121) * `dials.symmetry`: Prevent failures when dealing with small numbers of reflections (#1130, cctbx/cctbx_project#435)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The previous implementation held the reflection table up to 3 times in
memory during the output phase. This implementation passes the python
filehandle into C++ space and then writes the msgpack output directly
to the file. As a result memory consumption is significantly reduced.
Fixes #1112