-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce s3hook memory usage #37886
Reduce s3hook memory usage #37886
Conversation
If we think it's right to remove we can introduce a breaking change release. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a good start. I'd love to see Resources replaced with Clients some day to bring it in line with the other hooks, but that caching is nice and I like the cleanup work.
As far a I could see this PR do not replace boto3.resource by a boto3.client, in general it should reduce a bit of memory usage |
Closes #35449
Cached an S3 resource property for functions still using resources. Ran a simple loop of hook.get_bucket() for 5 buckets and captured memory data with memray. Memory usage went from 34.5mb without caching to 11.5mb with cached property. This PR doesn't address removing the use of Resources, since it would cause a breaking change for anyone directly calling
hook.get_bucket()
orhook.get_key()
.Cleaned up some of the other typing code and imports.