pytest: Selectively mark values as xfail in parametrize

11 June 2024

In these notes, we use the pytest parametrize decorator to apply marks to some values for that parameter, rather than marking the whole test.

Problem §

With pytest it’s easy to mark a whole test as xfail:

@pytest.mark.xfail(reason="not yet supported")
def test_something():
    ...

But if you are using parametrize, you may want to only mark specific values as xfail:

# TODO: Mark values above 80 as xfail, rather than failing.
@pytest.mark.parametrize("degrees", range(0, 90 + 1, 1))
def test_something(degrees):
    if degrees > 80:
        raise ValueError("not yet supported")
    ...

Solution §

Taking the example from above, the degrees input steps through all integer values from 0 to 90 inclusive.

If you want to mark values above 80 as xfail, the parameter can be rewritten as a function which yields parameters with optional marks:

def degrees_params():
    for v in range(0, 90 + 1, 1):
        marks = []
        if v > 80:
            marks.append(
                pytest.mark.xfail(
                    reason="unsupported after 80 degrees"
                )
            )
        yield pytest.param(v, marks=marks)

@pytest.mark.parametrize("degrees", degrees_params())
def test_something(degrees):
    ...

Output §

This outputs something like:

...
test/test_example.py::test_something[70] PASSED                                 [ 78%]
test/test_example.py::test_something[71] PASSED                                 [ 79%]
test/test_example.py::test_something[72] PASSED                                 [ 80%]
test/test_example.py::test_something[73] PASSED                                 [ 81%]
test/test_example.py::test_something[74] PASSED                                 [ 82%]
test/test_example.py::test_something[75] PASSED                                 [ 83%]
test/test_example.py::test_something[76] PASSED                                 [ 84%]
test/test_example.py::test_something[77] PASSED                                 [ 85%]
test/test_example.py::test_something[78] PASSED                                 [ 86%]
test/test_example.py::test_something[79] PASSED                                 [ 87%]
test/test_example.py::test_something[80] PASSED                                 [ 89%]
test/test_example.py::test_something[81] XPASS (unsupported after 80 degrees)   [ 90%]
test/test_example.py::test_something[82] XPASS (unsupported after 80 degrees)   [ 91%]
test/test_example.py::test_something[83] XPASS (unsupported after 80 degrees)   [ 92%]
test/test_example.py::test_something[84] XPASS (unsupported after 80 degrees)   [ 93%]
test/test_example.py::test_something[85] XFAIL (unsupported after 80 degrees)   [ 94%]
test/test_example.py::test_something[86] XFAIL (unsupported after 80 degrees)   [ 95%]
test/test_example.py::test_something[87] XFAIL (unsupported after 80 degrees)   [ 96%]
test/test_example.py::test_something[88] XFAIL (unsupported after 80 degrees)   [ 97%]
test/test_example.py::test_something[89] XFAIL (unsupported after 80 degrees)   [ 98%]
test/test_example.py::test_something[90] XFAIL (unsupported after 80 degrees)   [100%]

We can see that:

Tests up to 80 finish with PASSED, which is expected.
Tests from 81 to 84 finish with XPASS. This means we marked them as xfail, but they still passed.
Tests from 85 to 90 finish with XFAIL. This means we marked them as xfail, and they failed.

Conclusion §

Passing a function into parametrize which yields parameters gives us more flexibility for applying pytest marks. This enables us to selectively mark certain values.

In the above example, we could have just tested values up to 80 and avoided using xfail. However, testing up to 90 and using xfail for unsupported values enables us to see where the test actually starts to fail.

Tagged
Python
pytest