disable OOM kill and/or include swap

[artemisart]

Hello, awesome tool !
Would it be possible to disable killing on OOM or include swap space in the calculation before killing ? with something like --ignore-oom and --oom-include-swap maybe ? Thanks :)

[itamarst]

When you say "killing on OOM", what behavior are you seeing?

1. Fil has a mechanism to try to detect OOM and dump a report in advance, so you can see what happened.
2. Both Linux and macOS will kill -9 processes using too much memory.

What behavior would you like to see instead of observed behavior?

[artemisart]

Yes I'm talking about 1, fil try to detects OOM but ignore the swap space while doing so, which can be unwanted (at least I believe it should be configurable).

filprofiler/memapi/src/oom.rs

Lines 58 to 68 in dd9ccb1

	pub fn are_we_oom(&mut self, total_allocated_bytes: usize) -> bool {
	let available_bytes = self.memory_info.get_available_memory();
	// Check if we're in danger zone, with very low available memory:
	if available_bytes < self.minimal_required_available_bytes {
	eprintln!(
	"=fil-profile= WARNING: Available bytes {} less than minimal required {}",
	available_bytes, self.minimal_required_available_bytes
	);
	return true;
	}

For this let available_bytes should be self.memory_info.get_available_memory() + swap free space.

[itamarst]

I can try, but... both Linux and macOS have their own heuristics so the OS might kill things and then you get nothing. So in general I expect to be biased towards killing early. On the other hand, if your program would complete when run without Fil, killing early is definitely a bug. What OS are you on? Do you expect your program to complete without running out of memory when not using Fil?

[itamarst]

Given the OOM detection is a heuristic, and might always be wrong, the thing to do is probably add a --disable-oom-detection flag, to just allow opt out when it guesses wrong.

[itamarst]

Release 2021.7.0 is now out with this change; it should be up on PyPI within a few minutes, and on Conda-Forge by tomorrow, hopefully.

[artemisart]

I understand for the heuristic (well personally every server/laptop I had crawled for hours before doing on oom kill).

On the other hand, if your program would complete when run without Fil, killing early is definitely a bug. What OS are you on?

Yes it would have completed without fil, this is on a ubuntu 18.04 server. I can give you the fil logs if you need it, but basically I had 30GB ram, 98% used, and 60GB of swap nearly 100% free.
I believe you could use the heuristic (ram_available_bytes + swap_free) / (ram_total + swap_total) > 98%, maybe under anm additional --oom-include-swap.

Given the OOM detection is a heuristic, and might always be wrong, the thing to do is probably add a --disable-oom-detection flag, to just allow opt out when it guesses wrong.
Release 2021.7.0 is now out with this change; it should be up on PyPI within a few minutes, and on Conda-Forge by tomorrow, hopefully.

Awesome ! Thanks :)

@itamarst

[itamarst]

I was forced to yank the latest release due to #208 (the wheels are broken on some OSes in a way that wasn't caught by CI) so if you do new installs you will get older version for now until I get more tests and a bugfix out. If that's a problem let me know and I will post the wheels here for manual download.

[itamarst]

Release 2021.7.1 should be up on PyPI shortly.