[pgranger23]

Fifth step of following SPINE releases. A comparison of two SPINE tags needs more than the model:

• the commit each dump came from;
• whether the writer's default file layout changed;
• what the dumper saw but does not map.

dump_schema.py now records all three. The step after this one adds the comparison itself.

What changes

• New keys in the schema JSON. generate.py ignores them, so the generated C++ does not change.

  • spine_sha: the commit given with the new --spine-sha, which must be 40 hexadecimal characters. It is null without the option.
  • writer_format: HDF5Writer's format_name, legacy_format_version, current_format_version and supported_format_versions, plus default_format_version, the default of the format_version parameter of HDF5Writer.__init__.
    • Each is read so that a missing attribute or parameter gives null instead of an error.
    • A change of the writer's default layout shows here.
  • warnings: {kind, name, detail} entries:
    • unmapped_attribute: a FieldMetadata attribute outside MAPPED_ATTRIBUTES (the flags, plus return_type, length, dtype, units, enum, reference, reference_space and skip). At 1.4.0 that is only cat, the concatenation flag of the index fields.
    • unlisted_class: a class of spine.data with attr_metadata that is in neither DEFAULT_CLASSES nor the new EXCLUDED_CLASSES.

• EXCLUDED_CLASSES.

  • It holds the classes the old comment excluded: RecoFragment, TruthFragment, Neutrino and ImageMeta2D.
  • It also holds larcv's Particle, ParticleLabel (the compact record of cluster labels), and the bases DataBase and PosDataBase. spine.data exports all of these with attr_metadata at 1.4.0.
  • An alias of a listed class (Meta is ImageMeta3D) counts as listed.

• Errors. Any exception while importing or reading SPINE prints one line on stderr, dump_schema: <exception class>: <message>, writes nothing, and exits 1. An unknown --classes name is now a LookupError. Exit codes:

  Code	Meaning
  0	written
  1	SPINE could not be imported or read, or --strict and a type mismatch
  2	usage error

• schema/spine-1.4.0.json is re-dumped with --spine-sha 4726e068f17e433e2fec068dc8ee498fa9a0658f. Only the three keys and three notes are new.

• Docs:

  • tools/gen/README.md: the top-level keys, the error line, --spine-sha in the regeneration command, and the new keys in "What to check afterwards".
  • tools/gen/EGM.md:
    • the flags, the lists and the exit codes;
    • what the dumper stops on;
    • how the warnings are built;
    • "the model version names the schema it was generated from, not the newest SPINE it reads".

Checked

• The committed schema. Before the change, the dumper reproduces schema/spine-1.4.0.json byte for byte under LCG_110. After it, the dump differs only by the new keys and notes, and generate.py --check and make_member_list.py --check pass.

• Other tags. These are dumps into a scratch folder; v1.4.87, v1.4.94 and v1.4.96 are local tags in a scratch SPINE clone:

  Tag	Exit	Result
  v1.2.2, v1.2.3	0	writer format as at 1.4.0; one warning (cat); the SHA recorded
  v1.4.87: adds a class with field metadata to spine.data	0	unlisted_class TestRecord
  v1.4.94: renames the writer's format_version parameter	0	default_format_version null
  v1.4.96: deletes FieldMetadata.pdg	1	dump_schema: AttributeError: 'FieldMetadata' object has no attribute 'pdg'
  v0.16.3	1	dump_schema: AttributeError: type object 'RecoParticle' has no attribute 'attr_metadata'
  --classes RecoParticle NoSuchClass	1	dump_schema: LookupError: spine.data has no class NoSuchClass
  --spine-sha 4726e06	2	usage error

• Tests. ctest under LCG_110: 6 entries; 5 run and pass, and spine_reader_real_files is skipped without files.

• Lint. flake8 --max-line-length=100 on dump_schema.py reports only the two E203 warnings main already has.