gh-108388: Split test_multiprocessing_spawn #108396

vstinner · 2023-08-24T02:26:53Z

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.

Issue: Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration.

vstinner · 2023-08-24T02:37:13Z

Example running 4 slow tests:

./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest

Timing (reference => this PR): 3 min 22 sec => 2 min 22 sec (-1 min)

On the main branch:

10 slowest tests:
- test_multiprocessing_spawn: 3 min 22 sec
- test_asyncio: 2 min 10 sec
- test_multiprocessing_forkserver: 1 min 43 sec
- test_multiprocessing_fork: 1 min 17 sec

Total duration: 3 min 22 sec

With this PR + PR #108393 (needed for this test):

40 tests OK.

10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec

2 tests skipped:
    test_asyncio.test_windows_events test_asyncio.test_windows_utils

Total duration: 2 min 22 sec

All CPUs are well used, the system load peak is 12.33 which is a good thing (the machine has 12 threads/6 cores).

Full logs:

vstinner@mona$ ./python -m test test_asyncio test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_spawn -j0 --slowest
0:00:00 load avg: 6.17 Run tests in parallel using 14 child processes
0:00:01 load avg: 6.17 [ 1/42] test_asyncio.test_protocols passed
0:00:01 load avg: 6.17 [ 2/42] test_asyncio.test_futures2 passed
0:00:01 load avg: 6.17 [ 3/42] test_asyncio.test_buffered_proto passed
0:00:01 load avg: 6.17 [ 4/42] test_asyncio.test_context passed
0:00:01 load avg: 6.17 [ 5/42] test_asyncio.test_pep492 passed
0:00:02 load avg: 6.17 [ 6/42] test_asyncio.test_runners passed
0:00:02 load avg: 6.17 [ 7/42] test_asyncio.test_proactor_events passed
0:00:02 load avg: 6.17 [ 8/42] test_asyncio.test_queues passed
0:00:03 load avg: 6.17 [ 9/42] test_asyncio.test_server passed
0:00:03 load avg: 6.17 [10/42] test_asyncio.test_selector_events passed
0:00:03 load avg: 6.17 [11/42] test_asyncio.test_eager_task_factory passed
0:00:04 load avg: 6.80 [12/42] test_asyncio.test_locks passed
0:00:04 load avg: 6.80 [13/42] test_asyncio.test_threads passed
0:00:06 load avg: 6.80 [14/42] test_asyncio.test_timeouts passed
0:00:06 load avg: 6.80 [15/42] test_asyncio.test_transports passed
0:00:07 load avg: 6.80 [16/42] test_asyncio.test_streams passed
0:00:07 load avg: 6.80 [17/42] test_asyncio.test_windows_events skipped
test_asyncio.test_windows_events skipped -- Windows only
0:00:07 load avg: 6.80 [18/42] test_asyncio.test_base_events passed
0:00:07 load avg: 6.80 [19/42] test_asyncio.test_sslproto passed
0:00:07 load avg: 6.80 [20/42] test_asyncio.test_windows_utils skipped
test_asyncio.test_windows_utils skipped -- Windows only
0:00:08 load avg: 6.80 [21/42] test_asyncio.test_unix_events passed
0:00:09 load avg: 7.70 [22/42] test_asyncio.test_waitfor passed
0:00:10 load avg: 7.70 [23/42] test_asyncio.test_sock_lowlevel passed
0:00:10 load avg: 7.70 [24/42] test_asyncio.test_sendfile passed
0:00:11 load avg: 7.70 [25/42] test_asyncio.test_taskgroups passed
0:00:12 load avg: 7.70 [26/42] test_asyncio.test_futures passed
0:00:18 load avg: 8.68 [27/42] test_asyncio.test_ssl passed
0:00:28 load avg: 10.42 [28/42] test_multiprocessing_fork.test_threads passed
0:00:29 load avg: 11.02 [29/42] test_asyncio.test_subprocess passed
Unknown child process pid 313199, will report returncode 255
Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that handles pid 313199 is closed
0:00:32 load avg: 11.02 [30/42] test_multiprocessing_fork.test_manager passed -- running: test_asyncio.test_events (32.0 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310261) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313364) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:32 load avg: 11.02 [31/42] test_multiprocessing_forkserver.test_threads passed -- running: test_asyncio.test_events (32.9 sec)
0:00:34 load avg: 11.50 [32/42] test_multiprocessing_fork.test_misc passed -- running: test_asyncio.test_events (34.6 sec), test_asyncio.test_tasks (31.7 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=313266) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:40 load avg: 12.02 [33/42] test_asyncio.test_events passed (39.3 sec) -- running: test_multiprocessing_forkserver.test_misc (30.9 sec), test_asyncio.test_tasks (37.1 sec), test_multiprocessing_fork.test_processes (32.4 sec), test_multiprocessing_forkserver.test_manager (31.1 sec)
0:00:51 load avg: 12.33 [34/42] test_multiprocessing_spawn.test_threads passed -- running: test_multiprocessing_forkserver.test_processes (40.9 sec), test_multiprocessing_spawn.test_processes (32.8 sec), test_multiprocessing_forkserver.test_misc (42.1 sec), test_multiprocessing_spawn.test_misc (38.9 sec), test_asyncio.test_tasks (48.2 sec), test_multiprocessing_fork.test_processes (43.5 sec), test_multiprocessing_spawn.test_manager (39.7 sec), test_multiprocessing_forkserver.test_manager (42.2 sec)
0:00:51 load avg: 12.33 [35/42] test_multiprocessing_fork.test_processes passed (43.4 sec) -- running: test_multiprocessing_forkserver.test_processes (41.5 sec), test_multiprocessing_spawn.test_processes (33.5 sec), test_multiprocessing_forkserver.test_misc (42.7 sec), test_multiprocessing_spawn.test_misc (39.5 sec), test_asyncio.test_tasks (48.8 sec), test_multiprocessing_spawn.test_manager (40.4 sec), test_multiprocessing_forkserver.test_manager (42.9 sec)
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
/home/vstinner/python/main/Lib/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=310329) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
0:00:55 load avg: 12.06 [36/42] test_multiprocessing_forkserver.test_manager passed (45.5 sec) -- running: test_multiprocessing_forkserver.test_processes (45.0 sec), test_multiprocessing_spawn.test_processes (37.0 sec), test_multiprocessing_forkserver.test_misc (46.2 sec), test_multiprocessing_spawn.test_misc (43.0 sec), test_asyncio.test_tasks (52.3 sec), test_multiprocessing_spawn.test_manager (43.9 sec)
0:00:56 load avg: 12.06 [37/42] test_asyncio.test_tasks passed (52.9 sec) -- running: test_multiprocessing_forkserver.test_processes (46.3 sec), test_multiprocessing_spawn.test_processes (38.2 sec), test_multiprocessing_forkserver.test_misc (47.5 sec), test_multiprocessing_spawn.test_misc (44.3 sec), test_multiprocessing_spawn.test_manager (45.1 sec)
0:01:04 load avg: 10.75 [38/42] test_multiprocessing_forkserver.test_misc passed (54.3 sec) -- running: test_multiprocessing_forkserver.test_processes (53.8 sec), test_multiprocessing_spawn.test_processes (45.7 sec), test_multiprocessing_spawn.test_misc (51.8 sec), test_multiprocessing_spawn.test_manager (52.6 sec)
0:01:14 load avg: 9.63 [39/42] test_multiprocessing_spawn.test_manager passed (1 min 2 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 4 sec), test_multiprocessing_spawn.test_processes (56.1 sec), test_multiprocessing_spawn.test_misc (1 min 2 sec)
0:01:35 load avg: 12.00 [40/42] test_multiprocessing_spawn.test_misc passed (1 min 22 sec) -- running: test_multiprocessing_forkserver.test_processes (1 min 24 sec), test_multiprocessing_spawn.test_processes (1 min 16 sec)
0:01:52 load avg: 13.34 [41/42] test_multiprocessing_forkserver.test_processes passed (1 min 41 sec) -- running: test_multiprocessing_spawn.test_processes (1 min 34 sec)
0:02:22 load avg: 8.84 running: test_multiprocessing_spawn.test_processes (2 min 4 sec)
0:02:22 load avg: 8.84 [42/42] test_multiprocessing_spawn.test_processes passed (2 min 3 sec)

== Tests result: SUCCESS ==

40 tests OK.

10 slowest tests:
- test_multiprocessing_spawn.test_processes: 2 min 3 sec
- test_multiprocessing_forkserver.test_processes: 1 min 41 sec
- test_multiprocessing_spawn.test_misc: 1 min 22 sec
- test_multiprocessing_spawn.test_manager: 1 min 2 sec
- test_multiprocessing_forkserver.test_misc: 54.3 sec
- test_asyncio.test_tasks: 52.9 sec
- test_multiprocessing_forkserver.test_manager: 45.5 sec
- test_multiprocessing_fork.test_processes: 43.4 sec
- test_asyncio.test_events: 39.3 sec
- test_multiprocessing_fork.test_misc: 26.3 sec

2 tests skipped:
    test_asyncio.test_windows_events test_asyncio.test_windows_utils

Total duration: 2 min 22 sec

gpshead · 2023-08-24T15:43:44Z

Lib/test/_test_multiprocessing.py

@@ -6126,7 +6126,8 @@ class ThreadsMixin(BaseMixin):
 # Functions used to create test cases from the base ones in this module
 #

-def install_tests_in_module_dict(remote_globs, start_method):
+def install_tests_in_module_dict(remote_globs, start_method,
+                                 only_type=None, exclude_types=False):


This function needs proper documentation as to what it does and how it behaves. The new parameters behaviors are unobvious.

miss-islington · 2023-08-24T17:35:28Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

bedevere-bot · 2023-08-24T17:35:38Z

GH-108442 is a backport of this pull request to the 3.12 branch.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <[email protected]>

gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <[email protected]>

miss-islington · 2023-09-21T19:37:07Z

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <[email protected]>

bedevere-app · 2023-09-21T19:37:18Z

GH-109688 is a backport of this pull request to the 3.11 branch.

Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <[email protected]>

gh-108388: Split test_multiprocessing_spawn (GH-108396) Split test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn into test packages. Each package is made of 4 sub-tests: processes, threads, manager and misc. It allows running more tests in parallel and so reduce the total test duration. (cherry picked from commit aa9a359) Co-authored-by: Victor Stinner <[email protected]>

bedevere-bot mentioned this pull request Aug 24, 2023

Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

Closed

bedevere-bot added the awaiting core review label Aug 24, 2023

vstinner merged commit aa9a359 into python:main Aug 24, 2023
18 checks passed

bedevere-bot removed the awaiting core review label Aug 24, 2023

vstinner deleted the test_mp_packages branch August 24, 2023 03:50

vstinner mentioned this pull request Aug 24, 2023

gh-108388: test_concurrent_futures requires cpu #108389

Closed

gpshead reviewed Aug 24, 2023

View reviewed changes

gpshead added the needs backport to 3.12 bug and security fixes label Aug 24, 2023

bedevere-bot removed the needs backport to 3.12 bug and security fixes label Aug 24, 2023

vstinner added the needs backport to 3.11 only security fixes label Sep 21, 2023

bedevere-app bot removed the needs backport to 3.11 only security fixes label Sep 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-108388: Split test_multiprocessing_spawn #108396

gh-108388: Split test_multiprocessing_spawn #108396

vstinner commented Aug 24, 2023 •

edited by bedevere-bot

Loading

vstinner commented Aug 24, 2023

gpshead Aug 24, 2023

miss-islington commented Aug 24, 2023

bedevere-bot commented Aug 24, 2023

miss-islington commented Sep 21, 2023

bedevere-app bot commented Sep 21, 2023

gh-108388: Split test_multiprocessing_spawn #108396

gh-108388: Split test_multiprocessing_spawn #108396

Conversation

vstinner commented Aug 24, 2023 • edited by bedevere-bot Loading

vstinner commented Aug 24, 2023

gpshead Aug 24, 2023

Choose a reason for hiding this comment

miss-islington commented Aug 24, 2023

bedevere-bot commented Aug 24, 2023

miss-islington commented Sep 21, 2023

bedevere-app bot commented Sep 21, 2023

vstinner commented Aug 24, 2023 •

edited by bedevere-bot

Loading