Sam Hooke

GitLab CI and poetry-dynamic-versioning

The poetry-dynamic-versioning plugin enables configuring Poetry to automatically generate the package version number from your VCS (e.g. Git), rather than from the version field in pyproject.toml.

For example, you can tag a commit in Git, and the release built from that commit will use the name of the tag.

Using poetry-dynamic-versioning with GitLab §

The following steps assume you are building a Python wheel using Poetry and Tox.

  1. Install the plugin for Poetry:

    poetry self add "poetry-dynamic-versioning[plugin]"`
  2. Enable the plugin for your project:

    poetry dynamic-versioning enable
  3. In your .gitlab-ci.yml, set GIT_DEPTH: 0 for your build job:

      stage: build_wheel
        GIT_DEPTH: 0
        - tox -e py310-build
          - dist/*
  4. In your .gitlab-ci.yml, pass the current branch (CI_COMMIT_REF_NAME) into Tox as an environment variable:

      stage: build_wheel
        GIT_DEPTH: 0
        - MY_BRANCH=${CI_COMMIT_REF_NAME} tox -e py310-build
          - dist/*

    This assumes your tox.ini looks something like this:

    commands =
        # ...snip...
        poetry build
  5. In the tox.ini, pass through the MY_BRANCH environment variable:

    pass_env =
  6. Update pyproject.toml to specify pattern and format-jinja fields. The pattern is used to extract the version number from the tag, and the format-jinja is used to generate the version string. This is what I use:

    enable = true
    vcs = "git"
    pattern  = "^(?P<base>\\d+\\.\\d+\\.\\d+)(-?((?P<stage>[a-zA-Z]+)\\.?(?P<revision>\\d+)?))?"
    format-jinja = """
        {%- if distance == 0 -%}
            {{ base }}
        {%- elif env["MY_BRANCH"] == "main" -%}
            {{ bump_version(base) }}.alpha{{ distance }}+g{{ commit }}
        {%- else -%}
            {{ base }}.dev{{ distance }}+g{{ commit }}
        {%- endif -%}

That’s it!

If you’d like to know why GIT_DEPTH and MY_BRANCH are necessary, read on.

Ensuring poetry-dynamic-versioning works with GitLab §

Unfortunately, there are a couple of issues that can prevent poetry-dynamic-versioning from fully working with GitLab:

  • By default, GitLab performs a “detached HEAD” checkout, which can prevent the branch variable in format-jinja from working.
  • By default, GitLab performs a shallow clone, which may prevent other fields from working if the last tag was too many commits ago.

The details for these fixes are documented below.


The branch variable §

The branch variable is the name of the current branch, or if that fails, None:

branch (string or None)

This allows you to change the build version depending upon the branch. For example, to have all main builds use the version hello_from_main, and all branch builds have the version hello_from_branch, you would do:

format-jinja = """
    {%- if branch == "main" -%}
    {%- else -%}
    {%- endif -%}

This works locally, but unfortunately under GitLab CI the branch variable returns None, preventing it from working.

Verifying branch is None under GitLab CI §

You can verify this with the following:

format-jinja = """
    TEST_{{- branch -}}_TEST

This will fail locally and on GitLab CI, because TEST_<blah>_TEST is not a valid Python version. But crucially, it will fail differently!

Assuming you are running it on the branch main:

  • Locally it will fail with TEST_main_TEST.
  • On GitLab CI it will fail with TEST_None_TEST.

Investigating why we get None under GitLab CI §

Looking at the plugin source, the plugin gets the Version object from dunamai. Following through in the dunamai source, it gets the branch variable from Git:

code, msg = _run_cmd("git symbolic-ref --short HEAD", path, codes=[0, 128])
if code == 128:
    branch = None
    branch = msg

If we try this out manually in GitLab CI, we see the issue:

$ git symbolic-ref --short HEAD
fatal: ref HEAD is not a symbolic ref

The Git command is failing, which explains why branch is defaulting to None under GitLab CI.

It fails because Git is checking out a commit, rather than a branch, and so it has a “detached head”.

For example, see this GitLab tutorial:

Checking out 7226fc70 as detached HEAD (ref is main)...

Fixing the branch variable §

Since GitLab already knows the current branch, and poetry-dynamic-versioning provides the env object for reading from the environment, we can avoid the “deatched HEAD” issue by passing the current branch from GitLab into poetry-dynamic-versioning. Since Tox does not pass environment variables by default, it must be added into pass_env for this method to work.

Previously I used this solution, but it has some shortcomings.


The Dunamai README warns that it needs access to the full version history, and suggests using GIT_DEPTH: 0 in GitLab. This prevents GitLab from doing a shallow clone, and ensures that poetry-dynamic-versioning has enough information to populate variables such as distance.


Appendix §

Attempted fix of detached HEAD with checkout §

An alternate fix is to call the following in GitLab CI:

git checkout "$CI_COMMIT_REF_NAME"

Call it in before_script (or wherever makes more sense), and it will ensure the checkout does not have a detached HEAD. This will allow git symbolic-ref --short HEAD to get the branch name, and in poetry-dynamic-verisoning, the branch variable will be populated correctly.

This can however lead to issues where the wrong commit is used for the build!

Attempted fix with GIT_STRATEGY §

I attempted setting GIT_STRATEGY: clone to avoid the shallow clone issue, but it did not appear to be doing anything.

Debugging GitLab CI §

Using the CI_DEBUG_TRACE GitLab CI variable is very useful for debugging:

  CI_DEBUG_TRACE: "true"