Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect memory constraints during init. #36479

Merged
merged 1 commit into from
Jun 30, 2020
Merged

Conversation

maleadt
Copy link
Member

@maleadt maleadt commented Jun 30, 2020

I'm not sure #24617 should have been closed, as uv_get_total_memory doesn't change when running with, e.g., docker run --memory 3G. Instead, libuv now has uv_get_contrained_memory, which we should probably respect when initializing the heap.

I think this is the cause for some GPU CI issues, where docker kills the process once we exceed the memory limit (i.e. we don't get a nice OOM error). That happens often, I assume because the GC doesn't know about the actual memory limit.

Like nodejs/node#27508

@maleadt maleadt added GC Garbage collector backport 1.5 labels Jun 30, 2020
@maleadt maleadt requested review from vtjnash and yuyichao June 30, 2020 08:01
@maleadt
Copy link
Member Author

maleadt commented Jun 30, 2020

We should probably expose this somewhere in Sys too (or should Sys.total_memory report the constrained memory?), but I wanted to get this in 1.5 if deemed OK (or a patch release) so I left that feature out for now. This should be a strict improvement that only matters in a non-default setting, e.g., without doing anything:

julia> ccall(:uv_get_total_memory, Int64, ()) |> Base.format_bytes
"31.249 GiB"

julia> ccall(:uv_get_constrained_memory, Int64, ()) |> Base.format_bytes
"8192.000 PiB" (around typemax(Int64))

@JeffBezanson JeffBezanson merged commit 0a4f357 into master Jun 30, 2020
@JeffBezanson JeffBezanson deleted the tb/constrained_memory branch June 30, 2020 21:55
KristofferC pushed a commit that referenced this pull request Jul 8, 2020
@KristofferC KristofferC mentioned this pull request Jul 8, 2020
13 tasks
simeonschaub pushed a commit to simeonschaub/julia that referenced this pull request Aug 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
GC Garbage collector
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants