Memory management and Sobel Points

[jrhalket]

I'm getting memory issues on the HPC I use when I expand the number of Sobel points I use from e.g. 300 to 1000. Is there a reason why the call to TikTak would use appreciably more memory with more Sobel points? What is it storing?

[tpapp]

Currently, it stores all of them before deciding which to keep. Cf this line.

In principle this pass could be done online, ie always keeping just the best points.

That said, I am not sure this is the relevant cause of memory consumption, unless the dimension itself is very large, you are effectively just storing vectors, even with a dimension of 10 thousand you would use 10^4 * 8 * 1000 / (10^9) = 0.08 Gb of memory.

How large is your dimension?

[jrhalket]

Hi - my dim is much smaller: 23 parameters. This is why I was confused by the memory issue as just the vector of points shouldn't be an issue.

I only get the memory issue when the inner loop is large too. In other words, if I use 1000 sobel points and the inner loop problem == "small", then I don't get a memory issue. If I use 100 Sobel points and the inner loop == "large", I don't get a memory issue. But when the inner loop uses a lot of memory and I use a lot of Sobel points, then I get the memory issue.

[tpapp]

Do you use threading?

Are you sure there is no memory leak in your inner loop?

[jrhalket]

I use pmap.

And I'm not 100% sure there's no memory leak in the inner loop. I will check into this a bit more. Thanks

[jrhalket]

So I've checked into this a bit more and here's what I am finding:
I've tried with 500 Sobel points and everything works fine. After starting with those 500 points, it then subsequently evaluates the objective function more than 3000 times without a hitch. So if my objection function evaluations are leaking memory, it isn't causing a problem here.
When I go up to 1000 Sobel points I get some sort of issue. I don't change anything else in the program.
One random thing I'll mention: inside the objective function evaluation I have some println statements at the end of it that tell me what the fn and parameter values are. When TikTok iterates over the parameters it prints them out for each parameter evaluation. However it never prints this line during the part where it is computing the Sobel points.

[tpapp]

The printing interaction is just frompmap.

Can you try without parallelization though?

[jrhalket]

Thanks for helping. I am getting closer. The issue arises when I ask for more sobel pts than I ask the hpc for processors. I had asked for 501 processors. When I use 502 sobel pts I get a memory problem. I use pmap within the objective fn eval. But I notice in your code that maybe there is kind of automatic multithreading at the sobel-level?

…

On Sat, Jan 14, 2023, 2:33 AM Tamas K. Papp ***@***.***> wrote: The printing interaction is just frompmap. Can you try without parallelization though? — Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJGWDCH47KU6WSIGU63VWLWSJJFNANCNFSM6AAAAAATVQTI44> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tpapp]

Yes, that code predates composable parallel processing APIs. Please try with use_threads = false. Cf #36.