python-tests.yml

name: python-tests

# The cross-platform Python gate (issue #169).
#
# Before this job, GitHub CI ran 533 of the suite's ~3490 Python tests on Linux
# or macOS — and every one of them under BNGSIM_CODEGEN_JIT=mir on a
# -DBNGSIM_ENABLE_MIR=ON build. The number exercised in the DEFAULT configuration
# on any non-Windows host was zero. lint.yml runs no pytest by design;
# native-tests.yml is the C++ suite; wheels.yml runs only the cibuildwheel import
# smoke test; windows-tail.yml and windows-nfsim.yml are Windows-only. So a
# regression in SBML loading, .net parsing, events, steady state, SSA, conversion
# or coupling could only be caught on Windows, and only if the file happened to
# be named in one of two hand-maintained lists.
#
# The gap was structural, not a bug. Every job that runs pytest names a curated
# file list, so a NEW test file defaults to running nowhere, and the backstop was
# assumed to be the local pre-push hook — which covers one platform (whichever
# the developer is on, here macOS arm64) and which `git push --no-verify`
# removes. native-tests.yml stated that assumption outright.
#
# Three properties of this job are therefore deliberate, and each one is the
# thing that closes a specific half of the gap:
#
#   * NO paths filter. A gate that fires selectively reintroduces the per-file
#     opt-in through the trigger instead of through the run list. lint.yml takes
#     the same position for the same reason.
#   * NO file list. It runs `python/tests` as a directory, so a file added
#     tomorrow runs here with no workflow edit — the property native-tests.yml
#     gets from driving ctest rather than one named target.
#   * NO -D overrides. Every other workflow disables something (KLU off in
#     mir/windows-tail/windows-nfsim/native-tests, MIR on in mir.yml), which is
#     exactly how the shipped configuration ended up untested. Here the build
#     takes the pyproject defaults: KLU on (REQUIRE_KLU=ON), NFsim on, RuleMonkey
#     on, MIR off.
#
# It is the pre-push hook, on platforms the developer does not have. `uv sync
# --extra dev` provisions from uv.lock and the suite is the hook's own: the hook
# (.pre-commit-config.yaml, ci/pre_push_pytest.py) runs the same python/tests
# directory, split across pytest-xdist workers, so green here means what a
# clean `git push` means locally — on a host the developer does not have.
#
# NO leg installs SuiteSparse, and that is the point rather than an omission.
# All of them take BNGSIM_KLU_AUTOBUILD (GH #209) — the from-source fallback
# an sdist `pip install` on a bare box depends on — which no other job anywhere
# exercises: every wheel leg resolves a prebuilt SuiteSparse through
# SUITESPARSE_ROOT and every other job sets ENABLE_KLU=OFF.
#
# The bare Linux leg used to `apt-get install libsuitesparse-dev` because on a
# bare ubuntu-latest the autobuild did NOT work: SuiteSparse_config calls
# find_package(BLAS REQUIRED) whether or not anything links a BLAS, the runner
# image ships none, and the configure died there before KLU was reached
# (lanl/bngsim#178 — measured here, on this job's first run). The apt step was a
# workaround for that, so removing it is now the CI proof that #178 is fixed:
# ci/build_suitesparse.sh skips the probe outright, and a leg that installs
# nothing is exactly the bare host an sdist install lands on. Do not re-add it to
# fix a red build — a red build here means the self-sufficiency claim regressed.
#
# System-SuiteSparse discovery (the other half of GH #209) stays covered by
# cibuildwheel's Linux leg, which resolves `dnf install suitesparse-devel`
# through SUITESPARSE_ROOT.
#
# One measured side effect of dropping the apt step, so the skip count is not a
# mystery later: `libsuitesparse-dev` pulled in liblapack/libblas transitively,
# which is the only reason this leg ever built the optional LAPACK dense solver
# (GH #84). Without it `find_package(LAPACK QUIET)` finds nothing and seven tests
# go from passed to skipped on Linux — five "build links no BLAS dense backend"
# plus two "LAPACK-dense not built in this configuration" (85 -> 92 skips). That
# is the runner's real default state, not a loss: native-tests.yml already builds
# on a stock ubuntu-latest and reports "optimized dense solver DISABLED (no
# BLAS/LAPACK found)", so the Linux LAPACK build was an accident of the workaround
# and lasted exactly as long as it did. The seven still run on the macOS leg,
# where Accelerate supplies the backend. Installing liblapack-dev on THIS leg to
# get them back would put a BLAS on the host and silently retire the #178 proof
# above — do not do that.
#
# The seven tests are covered instead by a THIRD leg (GH #269), because leaving
# them to macOS alone was not good enough: Accelerate and reference LAPACK are
# the two implementations this integration is most likely to disagree about
# (symbol decoration, LP64 vs ILP64 default), and #265 left no job anywhere
# building a BLAS on Linux. That leg is `ubuntu-latest` + `liblapack-dev` and
# NOTHING else — still no SuiteSparse, so it still takes BNGSIM_KLU_AUTOBUILD on
# a host with no system SuiteSparse. It does not stand in for the bare leg and
# cannot retire its proof: the bare leg is still there, still installs nothing,
# and is still what an sdist install lands on. The two ubuntu legs differ by one
# package, which is the point — one is the wheel/bare-host configuration, the
# other is the source-install-on-a-LAPACK-box configuration that actually ships
# HAS_LAPACK_DENSE=True.
#
# native-tests.yml gained the matching C++ leg; between them the Ax=b kernel and
# the full-CVODE assertions (n_dense_blas_factorizations vs the Jacobian count,
# trajectory parity) both run against reference LAPACK now.
#
# The HAS_KLU assertion below is what keeps all three honest — without it a build
# that quietly lost KLU would skip the sparse tests and still report green. Each
# leg carries the same shape of assertion on HAS_LAPACK_DENSE, against the value
# THAT leg expects (false on bare ubuntu, true on macOS/Accelerate and on the
# liblapack leg), so neither a disappearing nor a re-appearing BLAS is silent.
#
# Two coverage axes are deliberately NOT taken here, so they are not mistaken for
# oversights:
#
#   * macOS x86_64. mir.yml already runs a macos-15-intel leg, and a
#     macOS-x86_64-specific *Python-level* regression is the least likely of the
#     three. Adding it is one line in the matrix.
#   * The Python floor. requires-python is >=3.10 and every job here runs 3.12,
#     so nothing tests the floor at runtime (ruff's target-version=py310 catches
#     syntax, not stdlib API). Adding it is one more matrix entry.
#
# Line coverage of bngsim's Python (issue #585) is measured on the ubuntu-latest
# leg only, both shards, and combined by the python-coverage job below: a report
# in that job's log and summary, never a gate. The other legs run the same tests
# unmeasured.
#
# One known skip is environmental rather than structural: $BNGPATH is unset, so
# the 14 BNG2.pl round-trip tests in test_sbml_to_bngl.py skip. Closing that needs
# the perl toolchain plus the `parity` dependency group (a git+https reference to
# PyBioNetGen), which is a heavier provisioning step than this gate warrants; the
# skip is declared in conftest's _DECLARED_SKIPS and shows up in the audit table.

on:
  push:
    branches: [main]
  pull_request:
  workflow_dispatch:
    inputs:
      pytest_args:
        description: "Extra pytest args (e.g. -x, -k some_pattern, or a single file path)"
        required: false
        default: ""

concurrency:
  group: python-tests-${{ github.ref }}
  cancel-in-progress: true

jobs:
  # Each leg is two jobs, one per shard of the suite (BNGSIM_TEST_SHARD below;
  # conftest collects each test file in one shard, balanced on
  # ci/test_durations.json). The
  # python-tests-counts job further down sums the two summaries of a leg for
  # the false-green guard, which is a statement about the whole suite.
  python-tests:
    name: Python suite · ${{ matrix.leg.label }} · ${{ matrix.shard }}/2
    runs-on: ${{ matrix.leg.os }}
    timeout-minutes: 90
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2]
        leg:
          - os: ubuntu-latest # x86_64 Linux — no Python coverage at all before #169
            label: ubuntu-latest
            apt: "" # bare host: the #178 proof. Do not add a package here.
            expect_lapack: "false"
            # The one leg that measures line coverage (#585). The python-coverage job
            # reads the source from a checkout at this leg's path, so keep it on ubuntu.
            coverage: "true"
          - os: macos-15 # macOS arm64; macos-14 retires 2026-11-02 (runner-images #13518)
            label: macos-15
            apt: ""
            expect_lapack: "true" # Accelerate, no install needed
            coverage: "false"
          # GH #269. Same bare-host build as the first leg plus exactly one
          # package, so the LAPACK dense solver (GH #84) is exercised against
          # reference LAPACK and not only against macOS Accelerate.
          - os: ubuntu-latest
            label: ubuntu-latest-lapack
            apt: liblapack-dev
            expect_lapack: "true"
            coverage: "false"
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/actions/sundials-archive

      # Only the LAPACK leg installs anything, and only this one package. The
      # other two legs must stay bare — see the header on lanl/bngsim#178.
      - name: Install liblapack-dev (LAPACK leg only)
        if: matrix.leg.apt != ''
        # A hung apt-get sat in this step for 79 minutes with no output on
        # lanl/bngsim#1198, and only the job's 90-minute limit would have ended
        # it. Each apt-get call now has 3 minutes, a failed or hung attempt is
        # tried once more, and the step as a whole has 15. apt itself waits up
        # to 2 minutes for a dpkg lock (unattended-upgrades on a fresh runner)
        # and retries a download 3 times before any of that applies.
        timeout-minutes: 15
        run: |
          set -euxo pipefail
          apt_install() {
            local opts=(-o DPkg::Lock::Timeout=120 -o Acquire::Retries=3)
            timeout --kill-after=30 180 sudo apt-get "${opts[@]}" update &&
              timeout --kill-after=30 180 sudo apt-get "${opts[@]}" install -y "$@"
          }
          apt_install ${{ matrix.leg.apt }} || {
            echo "::warning::apt-get failed or hung; trying once more"
            sleep 15
            apt_install ${{ matrix.leg.apt }}
          }

      - name: Install uv
        uses: astral-sh/setup-uv@v10.3.3
        with:
          prune-cache: true
          enable-cache: true
          cache-dependency-glob: "uv.lock"

      # `uv sync` below re-resolves instead of installing from the lock when the
      # lock does not match pyproject.toml, and a re-resolve fetches metadata for
      # every locked package — including the parity group's git-pinned
      # PyBioNetGen. The failure then surfaces as a PyBioNetGen build error on a
      # PR that never touched it (CONTRIBUTING.md says as much). `--check` asserts
      # the lock is up to date and costs milliseconds, so the real cause is named
      # here instead of being diagnosed from that misleading error later.
      #
      # It is deliberately tolerant of pure formatting differences between uv
      # versions. uv rewrites redundant dependency markers when it happens to
      # write the file, but `--check` does not treat that as stale, so this does
      # not fail on a lock written by a different uv.
      - name: Assert uv.lock is up to date
        run: uv lock --check

      # No SuiteSparse install step, on any leg — see the header. All three build
      # the pinned KLU subset from source through BNGSIM_KLU_AUTOBUILD, which is
      # what an sdist `pip install` on a bare host does.

      # --extra dev rather than an enumerated extras list: `dev` is the set a
      # developer's `uv sync` installs, so this job and the pre-push hook see the
      # same environment, and an extra added to `dev` later reaches CI without a
      # workflow edit. It is what un-skips the antimony / roadrunner / xarray /
      # h5py / pandas / jax / vivarium guarded tests — ~50 items that skip on a
      # bare install. Building bngsim itself is the slow part (SUNDIALS is
      # FetchContent'd and the KLU subset is built from source).
      #
      # ccache keeps that build's objects between runs. CMake (3.17+) takes
      # CMAKE_<LANG>_COMPILER_LAUNCHER from the environment at configure, which
      # reaches every CMake configure uv's build backend runs, the KLU autobuild
      # included. One cache per leg, written by shard 1 only: both shards build
      # the same thing, and two writers to one key would race. The key ends in
      # -build because the action restores by prefix: without it, the
      # ubuntu-latest leg's key is a prefix of the lapack leg's, and it restored
      # whichever of the two was saved last.
      #
      # ccache is a speed-up only, so a failure to set it up does not fail the
      # job: the build compiles through ccache only when it is installed. The
      # action installs it from apt or Homebrew, and on 2026-11-11 the Ubuntu
      # package servers stopped resolving for ten minutes, which failed every
      # Linux build leg here and in four other workflows (#1488).
      - name: Set up ccache
        continue-on-error: true
        uses: hendrikmuhs/ccache-action@v1.2
        with:
          key: python-tests-${{ matrix.leg.label }}-build
          save: ${{ matrix.shard == 1 }}
          max-size: 500M

      - name: Provision from uv.lock (default build config)
        run: |
          if command -v ccache >/dev/null; then export CMAKE_C_COMPILER_LAUNCHER=ccache CMAKE_CXX_COMPILER_LAUNCHER=ccache; fi
          uv sync --extra dev --python 3.12
          if command -v ccache >/dev/null; then ccache --show-stats; fi

      # libroadrunner's wheel links libpython by name; ubuntu-26.04 has no system
      # 3.12 to satisfy it. See ci/libpython_on_loader_path.sh.
      - name: Let RoadRunner find libpython
        run: bash ci/libpython_on_loader_path.sh

      - name: Show build / capability info
        env:
          EXPECT_LAPACK: ${{ matrix.leg.expect_lapack }}
        run: |
          set -euxo pipefail
          uv run --no-sync python -c "import platform, sys; print(platform.platform()); print(sys.version)"
          uv run --no-sync python -c "import bngsim; print('bngsim', bngsim.__version__); print('HAS_KLU', bngsim.HAS_KLU); print('HAS_NFSIM', bngsim.HAS_NFSIM); print('HAS_MIR', bngsim.HAS_MIR); print('HAS_LAPACK_DENSE', bngsim.HAS_LAPACK_DENSE)"
          # The point of this job is the DEFAULT configuration, and KLU is the
          # part of it nothing else in CI builds. A build that silently lost KLU
          # would skip the sparse-solver tests and still be green, so fail here
          # instead — the same false-green guard mir.yml puts on HAS_MIR.
          uv run --no-sync python -c "import bngsim, sys; sys.exit(0 if bngsim.HAS_KLU else 'HAS_KLU is False — the default build did not link SuiteSparse/KLU')"
          # Same guard for the BLAS dense backend (GH #269), in BOTH directions.
          # On the liblapack / macOS legs a False means the seven LAPACK tests
          # quietly turned back into skips and the leg stopped testing the thing
          # it exists to test. On the bare ubuntu leg a True means a BLAS arrived
          # on the host by some other route — which is the failure mode #265's
          # header warns about, since a BLAS is what made the #178 workaround
          # unnecessary to notice.
          uv run --no-sync python - <<'PY'
          import os
          import sys

          import bngsim

          expected = os.environ["EXPECT_LAPACK"] == "true"
          got = bngsim.HAS_LAPACK_DENSE
          assert bngsim.capabilities()["features"]["lapack_dense"] == got
          if got != expected:
              sys.exit(
                  f"HAS_LAPACK_DENSE is {got}, this leg expects {expected} — "
                  "the BLAS dense backend appeared or vanished on this host."
              )
          PY
          # The runtime codegen path shells out to this compiler at simulate time.
          uv run --no-sync python -c "from bngsim import _codegen; print('C compiler:', _codegen._find_c_compiler())"
          # --extra dev installs RoadRunner, and conftest declares a RoadRunner
          # skip as expected (it is a developer-only engine), so a wheel that
          # stops importing turns every test that needs it into a skip without a
          # word. ubuntu-26.04 did exactly that (libpython3.12.so.1.0 not found).
          uv run --no-sync python -c "import roadrunner; print('roadrunner', roadrunner.__version__)"

      # BNGSIM_SKIP_AUDIT=strict turns any skip whose reason is not in conftest's
      # _DECLARED_SKIPS into a failure. That is the second half of the false-green
      # guard below: the count floor catches tests that vanish from collection,
      # strict catches tests that quietly turn into skips for a reason nobody
      # signed off on.
      #
      # The suite runs this job's shard under pytest-xdist with work stealing:
      # one test at a time to whichever worker is free. One file per worker
      # (--dist loadfile) left test_expression_backend_equivalence.py, a fifth of
      # a shard's time, on a single worker. Every one of the 13,573 tests had
      # the same outcome both ways (2026-10-18). The workers share one artifact
      # cache: with -p no:cacheprovider each would otherwise take a temp dir of
      # its own (conftest's no-cacheprovider branch) and compile the same codegen
      # libraries again, and on macOS each new library also pays a first-load
      # check that the system runs one file at a time. The skip audit and the
      # summary line the counts job reads come from the xdist controller, as
      # they come from the one process serially.
      #
      # A dispatched run with pytest_args runs those args, unsharded, in shard 1.
      #
      # On the leg with `coverage` set, the shard also records which lines of
      # bngsim's Python ran (issue #585), into a data file the python-coverage
      # job combines with the other shard's. No report here: half the suite's
      # coverage is not a number anyone should read. COVERAGE_CORE=sysmon is
      # Python 3.12's sys.monitoring, which costs next to nothing where the
      # default tracer cost a test slice 70% more time (4.8 s plain, 5.0 s
      # sysmon, 8.3 s traced, measured on five test files). The interpreters the
      # tests spawn (`python -m bngsim.cache`, 49 call sites) are measured through
      # coverage's start-up hook, since pytest-cov sees only its own process.
      - name: Run the Python suite (this shard)
        id: pytest
        if: ${{ matrix.shard == 1 || !github.event.inputs.pytest_args }}
        env:
          BNGSIM_SKIP_AUDIT: strict
          BNGSIM_TEST_DATA: ${{ github.workspace }}/tests/data
          BNGSIM_TEST_CACHE_DIR: ${{ runner.temp }}/bngsim-test-cache
          BNGSIM_TEST_SHARD: ${{ matrix.shard }}/2
          PYTEST_ARGS: ${{ github.event.inputs.pytest_args }}
          COVERAGE_CORE: sysmon
          COVERAGE_FILE: ${{ github.workspace }}/coverage-${{ matrix.leg.label }}-${{ matrix.shard }}.dat
          COV_ARGS: ${{ matrix.leg.coverage == 'true' && '--cov=bngsim --cov-report=' || '' }}
          # Subprocesses the tests spawn are measured too (ci/subprocess.coveragerc
          # says how); empty on the other legs, which coverage's hook reads as unset.
          COVERAGE_PROCESS_START: ${{ matrix.leg.coverage == 'true' && format('{0}/ci/subprocess.coveragerc', github.workspace) || '' }}
        run: |
          set -o pipefail
          if [ -n "$PYTEST_ARGS" ]; then unset BNGSIM_TEST_SHARD; fi
          # shellcheck disable=SC2086  # COV_ARGS is two words or none
          uv run --no-sync python -m pytest -p no:cacheprovider python/tests \
            -n auto --dist worksteal $COV_ARGS \
            --junitxml=python-tests-${{ matrix.leg.label }}-${{ matrix.shard }}.xml \
            -q --tb=short --durations=15 ${{ github.event.inputs.pytest_args }} \
            2>&1 | tee python-tests-${{ matrix.leg.label }}-${{ matrix.shard }}.log

      # The parity harness's own unit tests (issue #316). Nothing ran these until
      # now, which is a false green over the code that decides every parity
      # verdict: `_core/differ.py` is the shared oracle for rr_parity,
      # bng_parity, copasi_parity and amici_parity alike, and a silent change
      # there does not produce a failing sweep — it produces a wrong one.
      #
      # This is NOT the sweeps. A sweep is hours against a gitignored corpus and
      # stays a local/manual job; this directory is unit tests of the harness
      # logic — schema, taxonomy, differ, outcome classification, parameter
      # alignment — and costs ~3 s. Every dependency it does not have it skips
      # for: `pytest.importorskip` covers amici and roadrunner, the BNG2.pl
      # resolver covers PyBioNetGen, and the corpus tests gate on the model file
      # rather than the directory (GH #192). Measured in a bare environment with
      # amici and roadrunner both unimportable: 420 passed, 23 skipped, 2.9 s.
      #
      # Run on every matrix leg rather than one, because `test_rr_common.py`
      # exercises the spawn-based worker scheduler, whose behavior is genuinely
      # platform-dependent.
      #
      # BNGSIM_SKIP_AUDIT is deliberately not set: the audit hook lives in
      # python/tests/conftest.py and its _DECLARED_SKIPS vocabulary does not
      # cover this suite, so `strict` here would fail on skips nobody has
      # triaged yet rather than catch anything.
      - name: Run the parity-harness unit tests
        id: parity_pytest
        if: ${{ !github.event.inputs.pytest_args }}
        run: |
          set -o pipefail
          uv run --no-sync python -m pytest -p no:cacheprovider parity_checks/tests \
            -q --tb=short 2>&1 | tee parity-tests-${{ matrix.leg.label }}-${{ matrix.shard }}.log

      - name: Guard against a false green (parity harness)
        if: ${{ !github.event.inputs.pytest_args }}
        run: |
          set -euo pipefail
          # Same two counts as the guard above, for the same two reasons, with
          # bounds in the same file. COLLECTED is environment-independent here
          # for a structural reason: every skip in this suite is function- or
          # class-level, so a missing engine moves tests between columns without
          # changing the total — 443 both fully provisioned and with amici,
          # roadrunner and BNG2.pl all absent, when this guard was added.
          LOG="parity-tests-${{ matrix.leg.label }}-${{ matrix.shard }}.log"
          summary="$(grep -E '[0-9]+ (passed|failed).* in [0-9.]+s' "$LOG" | tail -1 || true)"
          if [ -z "$summary" ]; then
            echo "::error::No pytest summary line — the parity suite did not run to completion."
            exit 1
          fi
          echo "summary: $summary"

          count() { echo "$summary" | grep -oE "[0-9]+ $1" | grep -oE '^[0-9]+' || echo 0; }
          passed="$(count passed)"
          skipped="$(count skipped)"
          collected=$(( passed + skipped + $(count xfailed) + $(count xpassed) ))

          bash ci/check_test_count.sh PARITY_COLLECTED "$collected" "parity tests stopped being collected rather than failing"
          bash ci/check_test_count.sh PARITY_SKIPPED "$skipped" "a group of parity tests turned into skips"

      # The SBML semantic test suite as a per-PR gate (issue #700). Before this
      # it ran on no pull request: the nightly (#702) alerts on a regression but
      # blocks nothing. The suite's expected values are its own, never bngsim's,
      # so unlike a golden-trajectory baseline this one cannot end up defending a
      # wrong answer bngsim once gave. Its 1,823 cases (1,789 scored) took about
      # 5 s on an M-series Mac.
      #
      # ci/sbml_suite_gate.py holds bngsim's outcome on every case to
      # parity_checks/nightly/baselines/sbml_semantic.json, the file the nightly
      # reads, and fails on ANY change. A good case that is not, a refusal that
      # became a wrong answer, or a case the harness could not score is for the
      # pull request to fix. An improvement is for it to record: re-run the
      # script with --update-baseline and commit the file, so a later pull
      # request that undoes the fix fails here.
      #
      # Every leg, not one: the outcomes are deterministic and the passing case
      # nearest its tolerance sat at 0.77 of it when this was added, so a leg
      # that disagrees is a platform-dependent answer worth seeing. It runs
      # whenever the pytest step ran, pass or fail, because an independent
      # oracle's verdict matters most when something else is already red.
      #
      # The corpus is not vendored. It is a sparse checkout of cases/semantic at
      # the commit in harness/sbml_test_suite/SUITE_PIN.json (~85 MB), and the
      # gate refuses (exit 2) a checkout at any other commit. The cache key is
      # the nightly's, so the ubuntu legs share its entry (the cache version
      # also hashes the path, which differs on macOS). actions/cache saves only
      # from a green job, so a red run re-fetches next time.
      - name: Read the SBML Test Suite pin
        id: sbml_pin
        if: ${{ !cancelled() && steps.pytest.conclusion != 'skipped' && !github.event.inputs.pytest_args }}
        run: |
          commit=$(python3 -c 'import json; print(json.load(open("harness/sbml_test_suite/SUITE_PIN.json"))["commit"])')
          echo "commit=$commit" >> "$GITHUB_OUTPUT"

      - name: Cache the pinned SBML semantic suite
        id: sbml_cache
        if: ${{ !cancelled() && steps.sbml_pin.outcome == 'success' }}
        uses: actions/cache@v4
        with:
          path: ${{ runner.temp }}/sbml-test-suite
          key: sbml-test-suite-semantic-${{ steps.sbml_pin.outputs.commit }}

      - name: Fetch the pinned SBML semantic suite (cache miss only)
        id: sbml_fetch
        if: ${{ !cancelled() && steps.sbml_pin.outcome == 'success' && steps.sbml_cache.outputs.cache-hit != 'true' }}
        run: uv run --no-sync python harness/sbml_test_suite/fetch_semantic_suite.py --sparse --dest "$RUNNER_TEMP/sbml-test-suite"

      - name: SBML semantic suite gate
        id: sbml_gate
        if: ${{ !cancelled() && steps.sbml_pin.outcome == 'success' && steps.sbml_fetch.outcome != 'failure' }}
        run: uv run --no-sync python ci/sbml_suite_gate.py --suite "$RUNNER_TEMP/sbml-test-suite" --fresh-out "sbml-suite-verdicts-${{ matrix.leg.label }}-${{ matrix.shard }}.json"

      # Conditioned on the pytest step having been reached, not just on always():
      # if the build fails there is no log to upload, and if-no-files-found=error
      # would then report a second, misleading failure on top of the real one.
      # Keeping `error` (rather than downgrading to `warn`) means a pytest step
      # that ran but produced no log is still caught.
      - name: Upload pytest log
        if: always() && steps.pytest.conclusion != 'skipped'
        uses: actions/upload-artifact@v4
        with:
          name: python-tests-log-${{ matrix.leg.label }}-${{ matrix.shard }}
          path: |
            python-tests-${{ matrix.leg.label }}-${{ matrix.shard }}.log
            python-tests-${{ matrix.leg.label }}-${{ matrix.shard }}.xml
          if-no-files-found: error

      - name: Upload parity pytest log
        if: always() && steps.parity_pytest.conclusion != 'skipped'
        uses: actions/upload-artifact@v4
        with:
          name: parity-tests-log-${{ matrix.leg.label }}-${{ matrix.shard }}
          path: parity-tests-${{ matrix.leg.label }}-${{ matrix.shard }}.log
          if-no-files-found: error

      - name: Upload coverage data
        if: always() && matrix.leg.coverage == 'true' && steps.pytest.conclusion != 'skipped'
        uses: actions/upload-artifact@v4
        with:
          name: python-coverage-${{ matrix.leg.label }}-${{ matrix.shard }}
          path: coverage-${{ matrix.leg.label }}-${{ matrix.shard }}.dat
          if-no-files-found: warn

      # This run's per-case verdicts, when they do not match the baseline. It is
      # the file --update-baseline would write, for a contributor without the
      # suite checked out locally.
      - name: Upload SBML suite verdicts
        if: always() && steps.sbml_gate.outcome == 'failure'
        uses: actions/upload-artifact@v4
        with:
          name: sbml-suite-verdicts-${{ matrix.leg.label }}-${{ matrix.shard }}
          path: sbml-suite-verdicts-${{ matrix.leg.label }}-${{ matrix.shard }}.json
          if-no-files-found: warn

  # Line coverage of bngsim's Python over the whole suite (issue #585): the two
  # shards of the coverage leg, combined. Reported, never gated: a minimum set
  # before anyone has read the baseline would be a no-op or a wall of red, and a
  # ratchet can follow once it is known. The full term-missing table is in this
  # job's log and artifact; the total goes to the run's summary page. It runs
  # when a shard failed too, since what ran is still worth seeing, and says so
  # when a shard's data is missing: half the suite reads as a coverage drop.
  python-coverage:
    name: Python coverage (report only)
    needs: python-tests
    if: ${{ !cancelled() && !github.event.inputs.pytest_args }}
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v10.3.3
        with:
          prune-cache: true

      - name: Download the coverage leg's shard data
        uses: actions/download-artifact@v4
        with:
          pattern: python-coverage-ubuntu-latest-?
          path: cov
          merge-multiple: true

      # The shards recorded absolute paths under this same checkout path, so the
      # report reads the source from this checkout. The version is uv.lock's.
      - name: Combine and report
        run: |
          set -euo pipefail
          shopt -s nullglob
          files=(cov/*.dat)
          if [ "${#files[@]}" -eq 0 ]; then
            echo "::warning::No coverage data from either shard: nothing to report."
            exit 0
          fi
          if [ "${#files[@]}" -ne 2 ]; then
            echo "::warning::Coverage data from ${#files[@]} of 2 shards: the total covers part of the suite."
          fi
          version="$(python3 -c 'import tomllib; print(next((p["version"] for p in tomllib.load(open("uv.lock", "rb"))["package"] if p["name"] == "coverage"), ""))')"
          if [ -z "$version" ]; then
            echo "::warning::coverage is not in uv.lock: nothing to report."
            exit 0
          fi
          cov() { uvx --from "coverage==$version" coverage "$@"; }
          cov combine --data-file=.coverage "${files[@]}"
          cov report --data-file=.coverage -m --skip-covered --sort=cover | tee coverage-report.txt
          cov xml --data-file=.coverage -o coverage.xml
          total="$(cov report --data-file=.coverage --format=total)"
          {
            echo "### Python line coverage (report only, issue #585)"
            echo ""
            echo "**${total}%** of bngsim's Python statements ran across the suite (ubuntu-latest, ${#files[@]} of 2 shard data files)."
            echo "The per-file table, with the missing lines, is in this job's log and the \`python-coverage-report\` artifact."
          } >> "$GITHUB_STEP_SUMMARY"

      - name: Upload the coverage report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: python-coverage-report
          path: |
            coverage-report.txt
            coverage.xml
          if-no-files-found: ignore

  # The false-green guard for a whole leg: the sum of its two shards' summaries.
  # conftest collects each test file in exactly one shard, so the sum is what
  # one unsharded job reported. The labels here are the legs' labels above;
  # test_ci_shards.py holds the two lists together. Like the per-job guard it
  # replaces, it runs only when every shard passed: a failed test is not in the
  # collected count, and the floor would add a misleading second error.
  python-tests-counts:
    name: Python suite counts · ${{ matrix.label }}
    needs: python-tests
    if: ${{ !github.event.inputs.pytest_args }}
    runs-on: ubuntu-latest
    timeout-minutes: 10
    strategy:
      fail-fast: false
      matrix:
        label: [ubuntu-latest, macos-15, ubuntu-latest-lapack]
    steps:
      - uses: actions/checkout@v4

      # `-?` and not `-*`: ubuntu-latest-* also matches ubuntu-latest-lapack-1.
      # Each artifact lands in a directory of its own name, so two can never
      # overwrite one another.
      - name: Download this leg's shard logs
        uses: actions/download-artifact@v4
        with:
          pattern: python-tests-log-${{ matrix.label }}-?
          path: logs

      - name: Guard against a false green
        run: |
          set -euo pipefail
          # A directory-scoped pytest cannot drift out of sync with a file list,
          # but it can still shrink: a module that stops importing skips at
          # collection, a rename orphans a file, a bad conftest deselects. None of
          # those is a failure — the run stays green with a smaller denominator,
          # which is the #28/#36 shape native-tests.yml guards with its own
          # floors. Two counts here, because there are two ways to shrink:
          #
          #   1. COLLECTED (passed + skipped + xfailed + xpassed) has a floor. A
          #      drop means tests stopped being collected. Every leg reports the
          #      same number, because a CI checkout has no gitignored corpora (a
          #      developer checkout that has them collects more; see
          #      ci/test_floors.env).
          #   2. SKIPPED has a ceiling. Crossing it means a group of tests turned
          #      into skips for a *declared* reason that BNGSIM_SKIP_AUDIT=strict
          #      accepts, e.g. an extra silently failing to install, provided
          #      the group is larger than the ceiling's headroom on that leg
          #      (30 to 56 tests; see ci/test_floors.env). This used to be a
          #      floor on PASSED, which had to keep moving as the suite grew; a
          #      skip count does not move when tests are added.
          #
          # The bounds live in ci/test_floors.env and ci/check_test_count.sh
          # holds each count to its own (issue #584). The check fails in both
          # directions: past the bound, and so far inside it that the bound has
          # gone stale. The second is what used to be missing. These floors had
          # sat untouched at 3480 and 3380 while the suite grew past 11000, so
          # two thirds of it could have vanished and this step stayed green.
          # "3 xpassed" cannot match "[0-9]+ passed", so the categories stay disjoint.
          #
          # A shard whose log has no summary did not finish, and the sum would be
          # short by a shard: fail on that rather than on the floor it would trip.
          collected=0
          skipped=0
          for shard in 1 2; do
            LOG="logs/python-tests-log-${{ matrix.label }}-$shard/python-tests-${{ matrix.label }}-$shard.log"
            summary="$(grep -E '[0-9]+ (passed|failed).* in [0-9.]+s' "$LOG" 2>/dev/null | tail -1 || true)"
            if [ -z "$summary" ]; then
              echo "::error::No pytest summary line for shard $shard/2 — it did not run to completion."
              exit 1
            fi
            echo "shard $shard/2: $summary"

            # "3 xpassed" cannot match "[0-9]+ passed", so the categories stay disjoint.
            count() { echo "$summary" | grep -oE "[0-9]+ $1" | grep -oE '^[0-9]+' || echo 0; }
            skipped=$(( skipped + $(count skipped) ))
            collected=$(( collected + $(count passed) + $(count skipped) + $(count xfailed) + $(count xpassed) ))
          done
          echo "leg total: $collected collected, $skipped skipped"

          bash ci/check_test_count.sh PYTEST_COLLECTED "$collected" "tests stopped being collected rather than failing"
          bash ci/check_test_count.sh PYTEST_SKIPPED "$skipped" "a group of tests turned into skips"