In these notes, we use the pytest parametrize
decorator to apply marks to some values for that parameter, rather than marking the whole test.
Problem § With pytest it’s easy to mark a whole test as xfail:
@pytest.mark.xfail ( reason = "not yet supported" )
def test_something ():
...
But if you are using parametrize
, you may want to only mark specific values as xfail:
# TODO: Mark values above 80 as xfail, rather than failing.
@pytest.mark.parametrize ( "degrees" , range ( 0 , 90 + 1 , 1 ))
def test_something ( degrees ):
if degrees > 80 :
raise ValueError ( "not yet supported" )
...
Solution § Taking the example from above, the degrees
input steps through all integer values from 0
to 90
inclusive.
If you want to mark values above 80
as xfail, the parameter can be rewritten as a function which yields parameters with optional marks:
def degrees_params ():
for v in range ( 0 , 90 + 1 , 1 ):
marks = []
if v > 80 :
marks . append (
pytest . mark . xfail (
reason = "unsupported after 80 degrees"
)
)
yield pytest . param ( v , marks = marks )
@pytest.mark.parametrize ( "degrees" , degrees_params ())
def test_something ( degrees ):
...
Output § This outputs something like:
...
test/test_example.py::test_something[70] PASSED [ 78%]
test/test_example.py::test_something[71] PASSED [ 79%]
test/test_example.py::test_something[72] PASSED [ 80%]
test/test_example.py::test_something[73] PASSED [ 81%]
test/test_example.py::test_something[74] PASSED [ 82%]
test/test_example.py::test_something[75] PASSED [ 83%]
test/test_example.py::test_something[76] PASSED [ 84%]
test/test_example.py::test_something[77] PASSED [ 85%]
test/test_example.py::test_something[78] PASSED [ 86%]
test/test_example.py::test_something[79] PASSED [ 87%]
test/test_example.py::test_something[80] PASSED [ 89%]
test/test_example.py::test_something[81] XPASS (unsupported after 80 degrees) [ 90%]
test/test_example.py::test_something[82] XPASS (unsupported after 80 degrees) [ 91%]
test/test_example.py::test_something[83] XPASS (unsupported after 80 degrees) [ 92%]
test/test_example.py::test_something[84] XPASS (unsupported after 80 degrees) [ 93%]
test/test_example.py::test_something[85] XFAIL (unsupported after 80 degrees) [ 94%]
test/test_example.py::test_something[86] XFAIL (unsupported after 80 degrees) [ 95%]
test/test_example.py::test_something[87] XFAIL (unsupported after 80 degrees) [ 96%]
test/test_example.py::test_something[88] XFAIL (unsupported after 80 degrees) [ 97%]
test/test_example.py::test_something[89] XFAIL (unsupported after 80 degrees) [ 98%]
test/test_example.py::test_something[90] XFAIL (unsupported after 80 degrees) [100%]
We can see that:
Tests up to 80
finish with PASSED
, which is expected. Tests from 81
to 84
finish with XPASS
. This means we marked them as xfail, but they still passed. Tests from 85
to 90
finish with XFAIL
. This means we marked them as xfail, and they failed. Conclusion § Passing a function into parametrize
which yields parameters gives us more flexibility for applying pytest marks. This enables us to selectively mark certain values.
In the above example, we could have just tested values up to 80
and avoided using xfail. However, testing up to 90
and using xfail for unsupported values enables us to see where the test actually starts to fail.