Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory usage can be bad #98

Open
andportnoy opened this issue Dec 21, 2018 · 0 comments
Open

memory usage can be bad #98

andportnoy opened this issue Dec 21, 2018 · 0 comments
Labels
optimization speeds things up, but isn't crucial

Comments

@andportnoy
Copy link
Collaborator

This issue groups three now closed memory-related issues.

  1. (improve memory usage with lmer #66)lmer fitting is memory expensive. At least three copies of the data are created when using multiple processes: in the parent process, in the child process, in the R process embedded in the child process. A solution is to use maxtasksperchild to purge R memory regularly. Another solution is to fit channel by channel, which has the same result.

  2. (chunk data when running in parallel #67) multiprocessing can't transfer objects > 2 Gb in size. Potential solution is to ensure the chunks are under 2 Gb.

  3. (memory #49) Objects can in general be very large. We can del certain attributes (like .data) and conserve a lot of memory.

@andportnoy andportnoy added the optimization speeds things up, but isn't crucial label Jan 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimization speeds things up, but isn't crucial
Projects
None yet
Development

No branches or pull requests

1 participant