Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3Boto3Storage still showing multithreading issues in 1.5.2 #268

Closed
tobiasmcnulty opened this issue Feb 19, 2017 · 0 comments
Closed

S3Boto3Storage still showing multithreading issues in 1.5.2 #268

tobiasmcnulty opened this issue Feb 19, 2017 · 0 comments
Labels
s3boto

Comments

@tobiasmcnulty
Copy link
Contributor

tobiasmcnulty commented Feb 19, 2017 •
edited
Loading

I'm using S3Boto3Storage in 1.5.2 with uWSGI & 8 threads, and my workers are exiting like so:

app_1    | DAMN ! worker 2 (pid: 15) died, killed by signal 11 :( trying respawn ... 
app_1    | Respawned uWSGI worker 2 (new pid: 20)

Unfortunately I'm not getting a stacktrace. I saw the bug fix that went into 1.5.2 in the release notes (#238) and, on a whim, switched to 1 thread per worker, and the segfaults went away. I also tried S3BotoStorage with multiple threads, and it works fine.

Happy to help debug this if I can, but I'm not entirely sure where to start.
@tobiasmcnulty tobiasmcnulty mentioned this issue Feb 19, 2017
Merged
2 tasks
tomkins added a commit to tomkins/django-storages that referenced this issue Jul 7, 2017
@tomkins
Create a session/connect per thread for s3boto3
71b7963 
Documentation for boto3 recommends a session per thread - https://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing

As the storage class is (usually) only instantiated once per process, we need to set a thread local for each thread/connection used.

Fixes jschneier#268
@tomkins tomkins mentioned this issue Jul 7, 2017
Merged
jschneier pushed a commit that referenced this issue Jul 27, 2017
@tomkins @jschneier
Create a session/connect per thread for s3boto3 (#358)
142e822 
Documentation for boto3 recommends a session per thread - https://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing

As the storage class is (usually) only instantiated once per process, we need to set a thread local for each thread/connection used.

Fixes #268
jschneier added a commit that referenced this issue Jul 27, 2017
@jschneier
Update CHANGELOG for #268 #358
6f8ef37
This was referenced Jul 27, 2017
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Closed
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
This was referenced Jul 31, 2017
Closed
Merged
Closed
Closed
Merged
Closed
@pyup-bot pyup-bot mentioned this issue Aug 7, 2017
Closed
@pyup-bot pyup-bot mentioned this issue Aug 14, 2017
Closed
This was referenced Aug 21, 2017
Closed
Closed
This was referenced Sep 1, 2017
Merged
Merged
Closed
@pyup-bot pyup-bot mentioned this issue Sep 11, 2017
Closed
@pyup-bot pyup-bot mentioned this issue Sep 18, 2017
Closed
@pyup-bot pyup-bot mentioned this issue Sep 25, 2017
Closed
This was referenced Oct 2, 2017
Closed
Merged
Closed
@pyup-bot pyup-bot mentioned this issue Oct 16, 2017
Closed
@pyup-bot pyup-bot mentioned this issue Oct 23, 2017
Merged
This was referenced Nov 10, 2017
Merged
Closed
@pyup-bot pyup-bot mentioned this issue Feb 11, 2018
Merged
nitely pushed a commit to satellogic/django-storages that referenced this issue Jul 30, 2018
@tomkins @nitely
Create a session/connect per thread for s3boto3 (jschneier#358)
3e2a867 
Documentation for boto3 recommends a session per thread - https://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing

As the storage class is (usually) only instantiated once per process, we need to set a thread local for each thread/connection used.

Fixes jschneier#268
nitely pushed a commit to satellogic/django-storages that referenced this issue Jul 30, 2018
@jschneier @nitely
Update CHANGELOG for jschneier#268 jschneier#358
8fa0dfa
nitely pushed a commit to satellogic/django-storages that referenced this issue Jul 30, 2018
@tomkins @nitely
Create a session/connect per thread for s3boto3 (jschneier#358)
11b2d92 
Documentation for boto3 recommends a session per thread - https://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing

As the storage class is (usually) only instantiated once per process, we need to set a thread local for each thread/connection used.

Fixes jschneier#268
nitely pushed a commit to satellogic/django-storages that referenced this issue Jul 30, 2018
@jschneier @nitely
Update CHANGELOG for jschneier#268 jschneier#358
5242182
nitely pushed a commit to satellogic/django-storages that referenced this issue Jul 30, 2018
@tomkins @nitely
Create a session/connect per thread for s3boto3 (jschneier#358)
03c707c 
Documentation for boto3 recommends a session per thread - https://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing

As the storage class is (usually) only instantiated once per process, we need to set a thread local for each thread/connection used.

Fixes jschneier#268
nitely pushed a commit to satellogic/django-storages that referenced this issue Jul 30, 2018
@jschneier @nitely
Update CHANGELOG for jschneier#268 jschneier#358
41f3060
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
s3boto
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tobiasmcnulty @jleclanche