Fixing unreasonable memory usage when creating an InferenceData object from an emcee sampler #2215
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The goal of this chang is to prevent large memory consumption during the creation of an InferenceData object from emcee. The change also improves efficiency.
OriolAbril
approved these changes
Feb 21, 2023
There was a problem hiding this comment.
left two minor comments about using from_emcee to keep the changelog user friendly (hopefully) as users don't need to know about io_emcee, also rename chain to samples_ary as that array contains samples from all chains.
Approving already though, that was a very good catch, thanks!
Codecov Report
@@ Coverage Diff @@
## main #2215 +/- ##
=======================================
Coverage 89.89% 89.90%
=======================================
Files 120 120
Lines 12423 12426 +3
=======================================
+ Hits 11168 11171 +3
Misses 1255 1255
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
|
Thanks ! :) |
bmwiedemann
pushed a commit
to bmwiedemann/openSUSE
that referenced
this pull request
Mar 22, 2023
https://build.opensuse.org/request/show/1073712 by user dgarcia + dimstar_suse - skip python 3.11 and python 3.8 because some dependencies doesn't support that python versions - Update to 0.15.1 - Fix memory usage and improve efficiency in `from_emcee` ([2215](arviz-devs/arviz#2215)) - Lower pandas version needed ([2217](arviz-devs/arviz#2217)) - 0.15.0 - Adds Savage-Dickey density ratio plot for Bayes factor approximation. ([2037](arviz-devs/arviz#2037), [2152](arviz-devs/arviz#2152)) - Add `CmdStanPySamplingWrapper` and `PyMCSamplingWrapper` classes ([2158](arviz-devs/arviz#2158)) - Changed dependency on netcdf4-python to h5netcdf ([2122](arviz-devs/arviz#2122)) - Fix `reloo` outdate
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The goal of this change is to prevent large memory consumption during the creation of an InferenceData object from emcee. The change also improves efficiency.
Previously, the full chain was initialized once for each of the variables, leading to long initialization time and a spike in memory usage (my 2.5Gb chain with ~30 dimensions led to 60+Gb of allocated memory, crashing the python process).
Checklist
📚 Documentation preview 📚: https://arviz--2215.org.readthedocs.build/en/2215/