[pgranger23]

Last step of following SPINE releases (stage 0 and stage 1). One command takes a SPINE tag through the whole chain and ends in one outcome:

tools/release/follow_spine.py v1.4.1

CI now runs it on the model's own tag on every push, and requires identical.

What changes

• tools/release/follow_spine.py (new, mode 100755, Python 3.10+). It runs these stages, writes status.json and one log per stage in an artifacts folder, and prints the command that reproduces the run:

  Stage	What it does
  s0_checks	tag form (vX.Y.Z; a pre-release or a tag below v1.2.1 is not followed), --apply's conditions, environment (numpy, h5py, yaml, scipy, git, CMake 3.21+, a compiler); no network
  s1_tag	git ls-remote and a checkout per commit. The model's own tag must still point at the schema's spine_sha
  s2_dump	dump_schema.py --spine-sha; the tag must equal v + spine.__version__
  s3_compare	identical when generate.generate() gives the same files for both schemas once the version is normalised, and type_check and writer_format are unchanged
  s4_fixtures	make_fixtures.py --tag: the tag's current set, written and read back by the tag itself, every cross-check
  s5_classify	classify.py. A replay (a --baseline other than the committed schema, or an older tag) stops here
  s6_build, s6_test	a git archive HEAD export with the new set (and, if changed, the new schema, regenerated model, check_names.py, golden_members.inc), CMake, ctest with --output-junit; then the regenerated spine-dump on HEAD's previous current_v1/current_v2, and a second classification
  s8_apply	--apply, after changed only: the schema, model, golden_members.inc, current set and manifest go into the clean working tree; nothing is committed

  The outcomes and exit codes:

  Exit	Outcome
  0	identical
  10	changed, with contracts_failed when spine_reader_contracts is the only failed entry
  20	not_followed
  30	environment_error, tag_moved, dump_failed, version_mismatch, generate_failed, fixture_tooling_failed, fixture_check_failed, build_failed, test_failed, internal_error
  2	usage error, with no status.json

• tools/release/classify.py (new) describes the change between two schemas, kind by kind:

  • fields (added with their position, removed, retyped, reordered, changed default, unit, flags, reference or doc);
  • classes, enums and scheme tables;
  • type_check and writer_format;
  • the decoded columns of the current set, where stored-width drift shows;
  • new warnings, compared by (kind, name) with the schema's and the committed manifest's (--baseline-warnings, an addition to the design so that fixture warnings are not new on every run), and substitutions;
  • added fields that the fixtures hold only at their default, and classes no fixture writes.

  It proposes the bump of the highest kind found, from one table: none < patch < minor < person, and "person" for any kind the table lacks. classification.md ends with the checklist of tools/gen/README.md.

• Tests, run by ctest's new spine_reader_release_tools entry for Python 3.10 or newer. They need neither SPINE nor the network:

  • test_classify.py mutates a frozen copy of the 1.4.0 schema (tools/release/tests/data/, not schema/) for every change kind;
  • it checks the precedence of the bumps and the real pair 1.2.2 → 1.2.3 (dumps of both tags made with the new dumper);
  • it checks that every change kind has a row in the bump table, and every row is reachable;
  • test_outcomes.py checks outcome_for(), check_usage() (including --apply on a dirty tree in a scratch git repository) and compare_schemas() (type_check alone, writer_format alone, a non-ASCII doc).

• CI.

  • ci/github-ci.sh --follow-spine DIR runs follow_spine.py v<schema version> after ctest and requires exit 0. The 1.14 job uses it.
  • When the job fails, it uploads the artifacts with actions/upload-artifact, pinned by SHA (v7.0.2).
  • The 1.12 job (Python 3.9) is unchanged.

• Fix: tools/gen/check_names.py failed on main. It scanned every file of src/model/, including EGM.md, whose "no I/O" matched the macro I of <complex.h>. It now scans only the .hpp and .cpp files. No ctest runs it, so nobody noticed; follow_spine.py runs it in s6_build.

• Docs:

  • tools/release/README.md (new): the environment, the stages, what to do for each outcome, replays, --apply, the bump table, and the scanner;
  • tools/release/EGM.md, tools/EGM.md, the root EGM.md and README: the new entry, the CI step, and --apply's side effects;
  • tools/gen/README.md: it points to follow_spine.py, and step 4 says to read classification.md first;
  • ci/EGM.md: the --follow-spine option and the dry-run command.

Checked

• Outcomes, from runs of follow_spine.py. The local tags are in a scratch clone of SPINE (--spine-repo <clone>):

  Tag	Change	Outcome
  v1.4.0 (GitHub, annotated)	none	identical (0)
  v1.4.99	adds RecoParticle.test_score	changed (10), patch; the follow-up says the fixtures hold it only at its default
  v1.4.98	csda_ke default	changed, contracts_failed, person
  v1.4.97	removes csda_ke	changed, contracts_failed, minor; the build passes
  v1.4.92	csda_ke unit	changed, contracts_failed, person
  v1.4.91	adds NuInteractionScheme.MARLEY with its tables	changed, patch, needs attention: "labels no fixture row stores: MARLEY"
  v1.4.90	swaps ParticleShape.SHOWER and TRACK	test_failed (30) at s6_test
  v1.4.88	removes crt_scores (no CAFMaker reads it)	changed, minor
  v1.4.87	adds a class with field metadata	identical, needs attention (unlisted_class)
  v1.4.86	HDF5Reader raises	fixture_check_failed
  v1.4.96	deletes FieldMetadata.pdg	dump_failed, reason AttributeError
  v1.4.95	version.py left at 1.4.0	version_mismatch
  v1.4.94	renames the writer's format_version	fixture_tooling_failed
  v1.4.93	the V2 writer drops one column	fixture_check_failed
  v1.4.0 moved to another commit		tag_moved
  v1.4.0rc1, v1.2.0		not_followed (20)
  1.4.0, v1.4, v1.4.80 (absent)		usage error (2), no status.json
  an unreadable --spine-repo		environment_error
  v1.2.3 --baseline tools/release/tests/data/spine-1.2.2.json		changed, replay, patch (enum additions)
  v1.2.1, v1.2.2		changed, replay; the three scheme substitutions listed

• --apply on v1.4.99, in a clean clone of this branch.

  • git status shows only the schema, the model, golden_members.inc, the current_* files and the manifest.
  • The manifest keeps v2root (v0.16.3) and v0.11.1_v1 (v0.11.1).
  • The applied tree builds, and its 7 ctest entries pass.
  • git checkout . && git clean -fd undoes it.
  • --apply with the model's own tag is a usage error.

• The identical run on v1.4.0 leaves git status clean.

• CI dry run in CERN's SFT images:

  • Ubuntu 24.04 with LCG_110 and --follow-spine: 7 entries, 6 run and pass. follow_spine.py v1.4.0 cloned SPINE from GitHub and gave identical in about 2.5 minutes.
  • Ubuntu 22.04 with LCG_105: 6 entries, 5 run and pass.

• Under LCG_110 on EL9: 7 ctest entries, 6 run and pass. check_names.py reports 0 clashes. flake8 --max-line-length=100 reports nothing on tools/release/.