Python: Support serverless environments #8061
Conversation
There was a problem hiding this comment.
This looks great @mr-brobot, thanks for working on this!
➜ python git:(serverless) hyperfine --warmup 1 "python3 /tmp/benchmark.py"
Benchmark 1: python3 /tmp/vo.py
Time (mean ± σ): 2.651 s ± 0.012 s [User: 3.020 s, System: 1.844 s]
Range (min … max): 2.637 s … 2.674 s 10 runs
➜ python git:(master) hyperfine --warmup 1 "python3 /tmp/benchmark.py"
Benchmark 1: python3 /tmp/vo.py
Time (mean ± σ): 2.670 s ± 0.029 s [User: 2.841 s, System: 1.934 s]
Range (min … max): 2.610 s … 2.711 s 10 runs
| func=_task_to_table, | ||
| iterable=[ | ||
| for table in executor.map( | ||
| lambda args: _task_to_table(*args), |
| # pylint: disable=redefined-outer-name,arguments-renamed,fixme | ||
| """Concurrency concepts that support multi-threading.""" | ||
| import threading | ||
| from concurrent.futures import Executor, ThreadPoolExecutor |
There was a problem hiding this comment.
@mr-brobot Thanks for raising this. Just to config, as I understand it, AWS Lambda does not support multiprocessing, but concurrent.futures.* is okay?
There was a problem hiding this comment.
@Fokko Yes! multiprocessing appears to rely on shared memory throughout the module, even in ThreadPool. concurrent.futures.ThreadPoolExecutor is recommended since it was "designed around threads from the start" which also suggests that there are some MP concepts entangled with ThreadPool.
There was a problem hiding this comment.
Got it, thanks for the explanation 👍🏻
|
Thanks for fixing this @mr-brobot, and looking forward to your use-case using AWS Lambda's |
Relates to #8011 and #8015
Tested on macOS, Ubuntu, and in AWS Lambda. 😄