Growing memory usage

[mike-burns]

Hi there. good_job is a cool project. I like the way it uses the Rails hooks.

Anyway, we've been seeing growing memory usage in our app and traced it down to good_job itself (we think). I created a fresh Rails app and only ran good_job, and watched the Resident Set Size (RSS) over 10 minutes. It increased, quickly at first, and then more slowly -- but it kept increasing.

Ruby: 3.1.0p0
Rails: 7.0.8.1
good_job: 3.26.1

My script (GNU/Linux):

#!/bin/sh

pid="$1"

while true; do
  printf "%s\t%skb\n" "$(date -Iseconds)" "$(ps -o rss= "$pid")"
  sleep 5
done

1. rails new example -d postgres && cd example
2. bundle add good_job
3. bin/rails db:create && bin/rails db:migrate
4. bundle exec good_job start

And then in another shell:

ps ax | grep good_job
./memory-watcher $that_pid > minimal-app-memory-use

(I know it's possible to get fancier but that wasn't the goal of the exercise.)

I've attached the full output to this ticket, but in summary:

2024-03-06T12:30:33-08:00	106808kb
2024-03-06T12:30:38-08:00	107024kb
2024-03-06T12:30:43-08:00	107084kb
2024-03-06T12:30:48-08:00	107272kb
2024-03-06T12:30:54-08:00	107300kb
2024-03-06T12:30:59-08:00	107692kb
2024-03-06T12:31:04-08:00	107720kb
2024-03-06T12:31:09-08:00	107940kb
2024-03-06T12:31:19-08:00	108156kb
2024-03-06T12:31:24-08:00	108184kb
2024-03-06T12:31:29-08:00	108444kb
2024-03-06T12:31:34-08:00	108472kb
2024-03-06T12:31:39-08:00	108724kb
2024-03-06T12:31:45-08:00	108780kb
2024-03-06T12:31:50-08:00	108968kb
2024-03-06T12:31:55-08:00	108996kb
2024-03-06T12:32:00-08:00	109376kb
2024-03-06T12:32:05-08:00	109400kb
2024-03-06T12:32:10-08:00	109612kb
2024-03-06T12:32:15-08:00	109652kb
2024-03-06T12:32:20-08:00	109844kb
2024-03-06T12:32:25-08:00	109876kb
2024-03-06T12:32:30-08:00	110212kb
2024-03-06T12:32:36-08:00	110240kb
2024-03-06T12:32:41-08:00	110424kb
2024-03-06T12:32:46-08:00	110452kb
2024-03-06T12:32:51-08:00	110652kb
2024-03-06T12:32:56-08:00	110684kb
2024-03-06T12:33:01-08:00	111016kb
2024-03-06T12:33:06-08:00	111200kb
2024-03-06T12:33:11-08:00	111220kb
2024-03-06T12:33:16-08:00	111416kb
2024-03-06T12:33:21-08:00	111440kb
2024-03-06T12:33:26-08:00	111648kb
2024-03-06T12:33:31-08:00	111808kb
2024-03-06T12:33:37-08:00	112012kb
2024-03-06T12:33:42-08:00	112036kb
2024-03-06T12:33:47-08:00	112228kb
2024-03-06T12:33:52-08:00	112260kb
2024-03-06T12:33:57-08:00	112452kb
2024-03-06T12:34:02-08:00	112616kb
2024-03-06T12:34:07-08:00	112816kb
2024-03-06T12:34:12-08:00	112856kb
2024-03-06T12:34:17-08:00	113016kb
2024-03-06T12:34:28-08:00	113188kb
2024-03-06T12:34:48-08:00	113448kb
2024-03-06T12:35:29-08:00	113712kb
2024-03-06T12:35:59-08:00	113976kb
2024-03-06T12:36:30-08:00	114240kb
2024-03-06T12:41:00-08:00	114504kb

Download the full 11-ish minutes (133 lines): minimal-app-memory-use.txt

[bensheldon]

That's weird! I can try setting that up and doing some heap dumps on that and give an explanation.

Here's some notes from my previous explorations if you wanted to try it yourself on your own app.: https://island94.org/2024/01/the-answer-is-in-your-heap-debugging-big-rails-memory

[bensheldon]

Hmm, I'm having trouble reproducing this. Albeit I'm using slightly different versions (Ruby 3.2.2, Rails 7.0.8.1)

I ran it for an hour and did some sampling and I only seem memory of:

409374016
409374208
409374656
409117776
409117776
409121968

When you were doing your sampling, were you running with RAILS_ENV=production, specifically code autoloading disabled / eager loading enabled? That would definitely affect your local debugging/profiling. (I ran my code in production mode).

But none of that is to disbelieve that you're seeing bad memory growth in production. I can totally believe that, though can't pinpoint the cause without more digging.

Here's my calendar if you wanted to look at it together: https://calendly.com/bensheldon/office-hours

My ask to prepare would be:

• Read the blog post about digging through the heap
• Installing derailed_benchmarks
• having a local bin/profile file (described in the blog post) with all of your environment variables/configuration to be able to run bin/profile rails s and have a version of your app boot locally

[mike-burns]

Thanks for offering to chat about this. I've set up bin/profile and scheduled some time.

I can confirm that with RAILS_ENV=production, the memory usage does level off. We are still seeing suspicious memory usage on the actual production job service so we might need help tracking it down.

[mike-burns]

Closing since the memory issue is likely elsewhere in our app. We did not successfully track it down, though.

[bensheldon]

Bummer that you couldn't track down the culprit ☹️