-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory management and Sobel Points #37
Comments
Currently, it stores all of them before deciding which to keep. Cf this line. In principle this pass could be done online, ie always keeping just the best points. That said, I am not sure this is the relevant cause of memory consumption, unless the dimension itself is very large, you are effectively just storing vectors, even with a dimension of 10 thousand you would use 10^4 * 8 * 1000 / (10^9) = 0.08 Gb of memory. How large is your dimension? |
Hi - my dim is much smaller: 23 parameters. This is why I was confused by the memory issue as just the vector of points shouldn't be an issue. I only get the memory issue when the inner loop is large too. In other words, if I use 1000 sobel points and the inner loop problem == "small", then I don't get a memory issue. If I use 100 Sobel points and the inner loop == "large", I don't get a memory issue. But when the inner loop uses a lot of memory and I use a lot of Sobel points, then I get the memory issue. |
Do you use threading? Are you sure there is no memory leak in your inner loop? |
I use pmap. And I'm not 100% sure there's no memory leak in the inner loop. I will check into this a bit more. Thanks |
So I've checked into this a bit more and here's what I am finding: |
The printing interaction is just from Can you try without parallelization though? |
Thanks for helping. I am getting closer. The issue arises when I ask for
more sobel pts than I ask the hpc for processors. I had asked for 501
processors. When I use 502 sobel pts I get a memory problem.
I use pmap within the objective fn eval. But I notice in your code that
maybe there is kind of automatic multithreading at the sobel-level?
…On Sat, Jan 14, 2023, 2:33 AM Tamas K. Papp ***@***.***> wrote:
The printing interaction is just frompmap.
Can you try without parallelization though?
—
Reply to this email directly, view it on GitHub
<#37 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJGWDCH47KU6WSIGU63VWLWSJJFNANCNFSM6AAAAAATVQTI44>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Yes, that code predates composable parallel processing APIs. Please try with |
I'm getting memory issues on the HPC I use when I expand the number of Sobel points I use from e.g. 300 to 1000. Is there a reason why the call to TikTak would use appreciably more memory with more Sobel points? What is it storing?
The text was updated successfully, but these errors were encountered: