-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add thread reaping to thread pool #702
Conversation
Seems like we should attack the initially problem, i.e. why threads in the pool are dying. I guess an exception is bubbling up to the top-level? |
I'm not completely sure what is killing the threads, from my educated guess, it's the fact that the exception is raised from another thread and thus bubbling up and killing the thread. All the issues we've linked above and also our initial problem we've faced is that heroku recently switched their recommended web server from unicorn to puma (ref), but due to the fact that puma does not have a request timeout |
@evanphx we are wondering what do you exactly mean with top level? |
rack-timeout is kind of terrible and currently doesn't play well with puma, I'm working on a bunch of changes to it, which should remedy that. But it seems to me threads could die for a number of valid reasons and puma should reap them when that happens. |
@kch I agree on the point that dead threads should be reaped, that is why we proposed this PR. On the other hand, could you maybe (in the rack-timeout repo) a bit more transparent on what kind a changes you are working and if you might need help? Happy to help :) Because of the issue mentioned we now switched back to unicorn. |
+1 for merge and release |
+1 |
2 similar comments
+1 |
+1 |
We'll switch back to unicorn too until this will be resolved: it's source of dangerous raising in request queueing during high request burst periods. |
Or you could run off of 1 thread and multiple puma workers which is essentially doing the same thing as Unicorn. |
Add thread reaping to thread pool
I had encountered the issue of request time-outs when I was testing my app using Any new updates on this? |
@acrolink same here. Sometimes I have only one thread working :( |
@emerleite .. If you don't mind trying a new framework I highy recommend Phoenix (upon Elixir) .. It is super fast compared to Ruby's Rails (5 or 6 times faster). As for Puma, I have kept using Unicorn for the meantime. |
We're currently doing that, but the Rails app must be running for now :) |
@acrolink Can you tell me more about your setup? |
Hi there.
The Problem
As already discussed in #691 and #681 Puma does not recover from dead threads. Even though this is a fairly uncommon state and might only happen in edge cases. One of these edge cases seems to be using rack-timeout as you can see in zombocom/rack-timeout#76 and rails/rails#1627 (comment).
To reproduce this behavior use the following example rack app:
Start it with
puma -w 1 -b tcp://localhost -p 9292 -t 3:3 dummy.ru
. And then force Puma to concurrently serve more requests than it has (healthy) threads available by using Apache Benchmarkab -n 8 -c 4 http://127.0.0.1:9292/
.Solution
This PR implements a thread reaping mechanism based on the auto trim functionality. It decreases the
@spawned
counter in theThreadPool
in order to allow it to spawn new threads.Alternative Solution
Instead you can use
th.abort_on_exception = true
in https://github.com/puma/puma/blob/master/lib/puma/thread_pool.rb#L112 which also solves the problem described above. We decided for the former solution because removing dead threads from the pool seemed a cleaner approach.As mentioned in #691 we are not sure if and how active record connections in dead threads will be cleaned up.
/cc @Partyschaum