Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gh-62432: unittest runner: Exit code 5 if no tests were run #102051

Merged
merged 13 commits into from
Apr 27, 2023
Merged

gh-62432: unittest runner: Exit code 5 if no tests were run #102051

merged 13 commits into from
Apr 27, 2023

Conversation

stefanor
Copy link
Contributor

@stefanor stefanor commented Feb 19, 2023 •
edited by bedevere-bot
Loading

As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7

It is common for test runner misconfiguration to fail to find any tests,
this should be an error.

Fixes: #62432

@bedevere-bot bedevere-bot mentioned this pull request Feb 19, 2023
Closed
@terryjreedy terryjreedy added type-feature A feature request or enhancement tests Tests in the Lib/test dir labels Feb 20, 2023
@terryjreedy
Copy link
Member

Tests/macOS is routinely failing; ignore currently.

This is a design bug fix, hence enhancement (feature change), as much as I would like to pretend otherwise.

hugovk
hugovk reviewed Feb 20, 2023
View reviewed changes
Lib/unittest/main.py Outdated Show resolved Hide resolved
Lib/unittest/runner.py Outdated Show resolved Hide resolved
Lib/test/test_unittest/test_program.py Outdated Show resolved Hide resolved
Lib/test/test_unittest/test_runner.py Outdated Show resolved Hide resolved
Misc/NEWS.d/next/Library/2023-02-19-12-37-08.gh-issue-62432.GnBFIB.rst Outdated Show resolved Hide resolved
Lib/test/test_unittest/test_program.py Outdated Show resolved Hide resolved
@stefanor stefanor requested review from hugovk and removed request for gpshead and ezio-melotti February 20, 2023 11:51
@stefanor
Copy link
Contributor Author

@stefanor stefanor requested review from hugovk and removed request for gpshead and ezio-melotti

Urgh, that removal wasn't intentional, I just hit the re-request review button.

@gpshead gpshead self-assigned this Feb 20, 2023
@stefanor
Copy link
Contributor Author

@gpshead: Ping? (it's been 2 months)

@gpshead gpshead enabled auto-merge (squash) April 27, 2023 01:04
@gpshead gpshead merged commit 76632b8 into python:main Apr 27, 2023
@stefanor stefanor deleted the fail-zero-tests branch April 27, 2023 11:49
@stefanor
Copy link
Contributor Author

Thank you!

@gpshead gpshead added the 3.12 bugs and security fixes label Jun 13, 2023
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
b274e14 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
@copybara-service copybara-service bot mentioned this pull request Sep 14, 2023
Merged
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
16bcbeb 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
4d8eba2 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
6c95d4d 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
9ff9bfa 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
8d1a40c 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
bbd341c 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
a4db4a8 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
14f99e8 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
609aa55 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
1edf2ab 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565771935
@zware zware mentioned this pull request Jan 3, 2024
Closed
@vstinner
Copy link
Member

vstinner commented Jan 8, 2024

The new Python 3.12 message surprises users when all tests are skipped: see issue #113661.

@fishy
Copy link

fishy commented Jan 8, 2024

The PR description says:

It is common for test runner misconfiguration to fail to find any tests

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.

@stefanor
Copy link
Contributor Author

stefanor commented Jan 9, 2024

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.

Yeah, I'd agree with that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gpshead gpshead gpshead approved these changes

@hugovk hugovk Awaiting requested review from hugovk

@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti

Assignees

@gpshead gpshead

Labels
3.12 bugs and security fixes tests Tests in the Lib/test dir type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

running a suite with no tests is not an error
9 participants
@stefanor @terryjreedy @vstinner @fishy @gpshead @hugovk @bedevere-bot @hauntsaninja @AlexWaygood