Sam Hooke

GitLab: Enable coverage reporting with pytest

GitLab supports coverage reporting, which displays the coverage percentage in the GitLab UI, e.g. in merge requests, jobs, and badges. Following are the steps for configuring pytest to generate coverage statistics, and hook them up to GitLab.

Update tox.ini to generate terminal and XML coverage §

Assuming your Python package is called my_package, update your pytest call in tox.ini to use --cov-report as follows:

tox.ini §
[testenv]
deps =
    pytest
    pytest-cov
commands =
    pytest \
        --cov-report=term \
        --cov-report=xml:coverage/{envname}/coverage.xml \
        --cov=my_package \
        tests

The important parts are:

  • --cov-report=term - to print the coverage out to stdout. This will be parsed by GitLab to display the total coverage percentage in the GitLab UI.
  • --cov-report=xml:<dir> - to write the coverage out in XML format. This will be used for detailed line-by-line coverage reports in the GitLab UI.

Update .gitlab-ci.yml to generate coverage artifacts §

Then, in .gitlab-ci.yml, use the coverage_report field with the following configuration to get GitLab to extract the coverage:

.gitlab-ci.yml §
unit_test:py310:
    stage: test
    artifacts:
        reports:
            coverage_report:
                coverage_format: cobertura
                path: 'coverage/py310/coverage.xml'
    coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'

The coverage_report is a special field used by GitLab to extract the coverage results.

The important parts are:

  • artifacts: ... reports: ... coverage_report: - this provides the coverage report to GitLab.
  • path: '<dir>' - this must be the same directory as in tox.ini.
  • coverage: <regex> - this defines the regex used to extract the total coverage percentage from stdout.

For example, your stdout might look like:

---------- coverage: platform linux, python 3.10.12-final-0 ----------
Name                                    Stmts   Miss  Cover
-----------------------------------------------------------
src/my_package/__init__.py                  0      0   100%
src/my_package/file_1.py                  102     12    88%
src/my_package/file_2.py                  128     59    54%
src/my_package/file_3.py                   24      4    83%
...
src/my_package/test/__init__.py             0      0   100%
src/my_package/test/test_1.py             103      0   100%
src/my_package/test/test_2.py              62      0   100%
src/my_package/test/test_3.py              47      0   100%
...
-----------------------------------------------------------
TOTAL                                     987    145    85%
Coverage XML written to file coverage/py310/coverage.xml
=================== 98 passed, 7 skipped in 77.14s (0:01:17) ===================
  py310: OK (160.28=setup[71.12]+cmd[9.29,79.88] seconds)
  congratulations :) (160.58 seconds)

Then the regex is extracting the line starting with TOTAL.1

Bonus: separate coverage directory for each Python version §

We can use $(echo $CI_JOB_NAME | cut -d : -f 2) to extract the Python version from the job name. For example, this will convert unit_test:py310 to py310.

This can be useful if you wish to have a separate coverage directory for each Python version:

.gitlab-ci.yml §
.unit_test_template: &unit_test_template
    stage: test
    artifacts:
        reports:
            coverage_report:
                coverage_format: cobertura
                path: 'coverage/$(echo $CI_JOB_NAME | cut -d : -f 2)/coverage.xml'
    coverage: '/TOTAL\s+\d+\s+\d+\s+(\d+%)/'

unit_test:py310:
    <<: *unit_test_template
    script:
        - tox -e py310

unit_test:py311:
    <<: *unit_test_template
    script:
        - tox -e py311

  1. The GitLab documentation includes many example regexes, including one for pytest, but it did not manage to match for my pytest output. ↩︎