Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky re-runs xfail tests #184

Open
nedbat opened this issue Nov 18, 2021 · 0 comments
Open

Flaky re-runs xfail tests #184

nedbat opened this issue Nov 18, 2021 · 0 comments

Comments

@nedbat
Copy link

nedbat commented Nov 18, 2021

If I mark a test as xfail (expected to fail), then flaky shouldn't re-run the test when it fails. Additionally, xfail tests are counted twice, while regular failing tests are counted once.

test_it.py:

import pytest

@pytest.mark.xfail
def test_xfail():
    assert False

def test_fail():
    assert False

Run with pytest:

% pytest -vv --force-flaky
============================================ test session starts ============================================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/virtualenvs/tmp-d591713aeced424/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/flakybug
plugins: flaky-3.7.0
collected 2 items

test_it.py::test_xfail XFAIL                                                                          [ 50%]
test_it.py::test_xfail XFAIL                                                                          [ 50%]
test_it.py::test_fail FAILED                                                                          [100%]

================================================= FAILURES ==================================================
_________________________________________________ test_fail _________________________________________________

    def test_fail():
>       assert False
E       assert False

test_it.py:8: AssertionError
===Flaky Test Report===

test_xfail failed (1 runs remaining out of 2).
	<class 'AssertionError'>
	assert False
	[<TracebackEntry /private/tmp/flakybug/test_it.py:5>]
test_xfail failed; it passed 0 out of the required 1 times.
	<class 'AssertionError'>
	assert False
	[<TracebackEntry /private/tmp/flakybug/test_it.py:5>]
test_fail failed (1 runs remaining out of 2).
	<class 'AssertionError'>
	assert False
	[<TracebackEntry /private/tmp/flakybug/test_it.py:8>]
test_fail failed; it passed 0 out of the required 1 times.
	<class 'AssertionError'>
	assert False
	[<TracebackEntry /private/tmp/flakybug/test_it.py:8>]

===End Flaky Test Report===
========================================== short test summary info ==========================================
FAILED test_it.py::test_fail - assert False
======================================= 1 failed, 2 xfailed in 0.06s ========================================

The xfail test is listed twice in the -vv lines, and counted twice in the summary at the end.

% pip freeze
attrs==21.2.0
flaky==3.7.0
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.6
pytest==6.2.5
toml==0.10.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant