[Montana]

Fixes #1653.

Important

A .net file holding a byte that is not UTF-8 failed with a bare UnicodeDecodeError that named neither the file nor the line. A Latin-1 species name (Aµ(), the byte 0xb5) loaded, and its name broke on the way to Python. A refusal quoting such a byte lost its words the same way. The loader now refuses a line it reads that is not UTF-8, with the ModelError and ValueError it documents, quoting the line with the byte shown as \xb5. Every other refusal, the table-function reader's among them, shows such a byte the same way. write_net now writes UTF-8 whatever the locale.

The problem

pybind11 hands every name, expression and message the core makes to Python through Python's strict UTF-8 decoder, and the loader never checked its bytes against it. On main (80b15ae4), with the issue's script and the neighbouring routes:

file	main	here
Latin-1 Aµ() in the species block	bare UnicodeDecodeError from from_net and parse_net_file	ModelError / ValueError: species line '1 A\xb5() 3' holds the byte 0xb5, which is not UTF-8
the issue's refused reaction line (1 1 2 k 7 µ in Latin-1)	the refusal's words replaced by the decode error	refused, quoting the line with \xb5
a Latin-1 byte in a line outside any block, a block's name, an end marker	the decode error in place of each refusal	each refusal's own words, the byte shown as \xb5
a Latin-1 byte in a tfun table file's data row or header	bare UnicodeDecodeError, from both loaders and build_model_from_parsed	the table reader's own refusal, the byte shown as \xb5
a bare-CR file with a run of 0xb5 bytes across #1511's 300-character cut	the decode error; each byte read as a continuation, so the cut took the whole run	603 characters for any run length
a byte in a comment, a block passed over, or an action line	loads	loads, unchanged
write_net of a UTF-8 Aµ() model under a Latin-1 locale	writes the byte 0xb5, which no loader can read back	writes UTF-8, and it round-trips
write_net of AΩ() under a Latin-1 locale	UnicodeEncodeError	writes UTF-8, and it round-trips

The referees on the Latin-1 file: run_network 3.0 reads it, to A(0.4) = 1.347987 (the closed form 3 e^-0.8). BNG2.pl 2.9.3's readFile refuses it ("Invalid Molecule name"), and refuses the UTF-8 one too, since its names are ASCII. So the refusal is stricter than run_network and agrees with BNG2.pl, and the UTF-8 file keeps loading here as it did.

The fix

Important

The issue suggests a UTF-8 check of the whole file. That would refuse six kinds of file that load on main, and whose bytes never reach Python: a byte in the header comment, a whole-line comment, a reaction's or a species' trailing comment, a block passed over, or an action line. So the check is on the text the loader reads: each of the five block parsers checks its line with the comment off, the same text it then parses. A control pins each of the six.

include/bngsim/utf8.hpp (new, header-only):

• utf8_char_length reads the well-formed sequences of the Unicode Standard's Table 3-7, which are the ones Python's strict decoder takes: no overlong form, no surrogate, nothing past U+10FFFF.
• shown_as_utf8 shows each byte outside them as \xhh, as Python's backslashreplace does, and leaves UTF-8 text unchanged.

src/net_file_loader.cpp:

• require_utf8, called by each of the parameters, species, functions, reactions and groups parsers after its end-of-block test, refuses a line that is not UTF-8. It uses the parser's own label (species line, functions line, ...).
• parse_net_file_structure's wrapper shows any such byte in a refusal as \xhh before A .net file with CR line endings loads as an empty model: 0 species and 0 reactions for 2 and 2, with no error (run_network refuses it) #1511's bare-CR handling. Then the 300-character cut runs over UTF-8 text, where a continuation byte always belongs to a character, so it stays at most three bytes past the mark.

src/table_function.cpp: the four refusals that quote the table file (a data line, either header column, a one-column header) show it as UTF-8. The table is read when the model is built, outside the loader's wrapper, and build_model_from_parsed reads it through no wrapper at all, so the reader is where it is fixed.

python/bngsim/convert/_net_writer.py: write_net writes UTF-8, which the loader reads, rather than the locale's encoding. Under a Latin-1 or cp1252 locale (Windows) it wrote Aµ() as a byte the loader refuses, and could not write AΩ() at all.

Docs and CI:

• Model.from_net's Raises and the loading guide name the refusal, and the guide says what still loads.
• windows-tail.yml runs the new test file. That leg already fires on src/net_file_loader.cpp, and Windows, at cp1252, is where write_net's half of this lives.
• conftest.py declares the new test's one skip, for a platform whose C locale is UTF-8, which none of CI's is.

How this was checked

1. Reproduced on main with the issue's script, and every row of the table above with probes of the neighbouring routes.
2. Against Python's own decoder: 9,312 byte strings in a species name. These are every lone high byte (128), every pair of 24 bytes at the edges of Table 3-7 (576), triples of them over every third third byte (4,608), and 4,000 random runs mixing real characters with edge bytes.
   • A name loads exactly when its bytes decode, and every one loads to the decoded name.
   • Every refusal names the byte Python's error starts at, and quotes the line exactly as backslashreplace shows it.
   • 0 disagreements. The test file holds the deterministic part, 722 strings, as one case.
3. Nothing that loaded before moves: every unique .net I have, 178 files (the repository's plus local copies, 36 of them holding bytes over 0x7f), on main and here, on the final tree.
   • 178 of 178 give the same model and the same parse_net_file dictionary: species, reactions, parameters, frozen_params and a digest of the whole dictionary.
4. Cost: Model.from_net on fceri_gamma.net (2.4 MB, 58,276 reactions), median of 7, two rounds each: 95.1 / 95.0 ms on main, 94.3 / 97.2 ms here, inside the rounds' own spread.
5. New tests fail on main for the right reason: test_net_not_utf8.py, 28 cases.
   • 20 fail on main, each with the UnicodeDecodeError, or write_net's UnicodeEncodeError under the ASCII locale:
     • the issue's file;
     • one per block;
     • the three refusals outside the blocks;
     • the CR cut at three run lengths;
     • the decoder comparison;
     • four table-file refusals;
     • build_model_from_parsed;
     • write_net for µ and Ω.
   • 8 controls pass on main: a UTF-8 name loads and runs to the closed form, the six bytes nothing reads, and a byte in a table file's comment.
6. Mutation tests: 16 mutants against the new file and A .net file with CR line endings loads as an empty model: 0 species and 0 reactions for 2 and 2, with no error (run_network refuses it) #1511's, all killed:
   • the check never refusing (7 failing);
   • the functions block unchecked (1), the groups block unchecked (1);
   • the comment checked too (1);
   • surrogates taken (1), an overlong E0 taken (1), past U+10FFFF taken (1), C0 and C1 taken (1), the third and fourth bytes unchecked (1);
   • refusals not escaped (6);
   • the CR cut on the raw words (3);
   • each of the table reader's four refusals left raw (2, 1, 1, 1);
   • write_net in the locale's encoding (2).
7. C++ suite: ctest, all binaries pass.
8. Special legs over the 363 test files that load, write or convert a .net, on the final tree:
   • BNGSIM_SKIP_AUDIT=strict: 13,928 passed, 84 skipped;
   • forced cc (BNGSIM_CODEGEN_THRESHOLD=1): 13,914 passed, 98 skipped;
   • forced MIR: 13,885 passed, 127 skipped.
9. Full Python suite, macOS arm64, no corpus (as CI runs it), on 80b15ae4, on the final tree: 19,576 passed, 140 skipped, 1 xfailed, 0 failed.
   • Collected floor: 19,717 collected, 266 above PYTEST_COLLECTED_FLOOR (19,451), inside the 500 slack.
   • Skips: 140, under PYTEST_SKIPPED_CEILING (252). The new file's one skip fires on no CI platform.
10. Docker, Linux x86-64 (python:3.12, --platform linux/amd64, gcc 14, built from the final tree with the [antimony,hdf5] extras and scipy):
    • The C locale there, with coercion off: ANSI_X3.4-1968 (ASCII), so the write_net test runs rather than skips.
    • The two loader test files: 58 passed.
    • Full suite: 19,012 passed, 537 skipped, 1 xfailed, 4 failed. The 4 are this container's standing failures (test_codegen_compile_reaping, test_codegen_toolchain_verdict, test_run_tests_sh, test_vendor_script_git_cwd), which fail the same way on main in it and on simulator: leave assignment-rule species at the value a run reports (#1510) #1637's to eigenvalues: isolate rows and columns before balancing, as dgebal does (#1529) #1645's runs; none touches the loader.
11. pre-commit --all-files passes (13 hooks, clang-format, ruff and mypy among them). ci/changelog.py check passes, and local sphinx-build -W exits 0.

Which CI legs this PR runs

The MIR and Windows legs run only on the files in their paths: filters. This PR changes the loader, the table reader, a new header, write_net, _model.py's docstring, a test file, conftest.py, the loading guide, windows-tail.yml and a changelog fragment.

leg	runs here?	what covers it
python-tests.yml (Ubuntu, macOS)	yes	the whole suite, the new file included
native-tests.yml	yes (src/**)	ctest, locally on the final tree
windows-tail.yml (MSVC)	yes, through src/net_file_loader.cpp, _model.py and its own file	the loader built under MSVC, and the new test file, now on its run list, at Windows' cp1252
mir.yml	no	forced MIR locally, over the files that load a .net (step 8)
windows-klu.yml, windows-nfsim.yml	no	not run: their tests target the KLU and NFsim paths. The reader is built and its tests run under MSVC by the leg above

Checked against past review findings

• Sibling sites (the top review finding):

  • both .net loaders, Model.load and net_to_sbml, which go through them;
  • the table-function reader, on three routes;
  • write_net, the loader's twin.

  Each is pinned. sbml_to_net's temporary .net is written in the locale's encoding too, but its text is ASCII, even from an SBML model with Unicode names (measured), so it is left alone.

• Refusing what main gets right (cvode: confirm a state switch's order with an event against a tighter run (#1025) #1146):

  • the six kinds of byte nothing reads keep loading, each pinned;
  • a UTF-8 name keeps loading;
  • the corpus does not move.

• Read as the engine reads: the UTF-8 rule is Python's own, and 9,312 strings agree with its decoder.

• A control must check the effect, not the report (nfsim: refuse a seed past 2**32 - 1, which NFsim ran as its low word (#1515) #1640): the controls check the loaded species and the trajectory, and the write_net test reloads what it wrote.

• Mutate the neighbours (net: refuse a species index that is not its line (#1512) #1639, nfsim: refuse a seed past 2**32 - 1, which NFsim ran as its low word (#1515) #1640): a loader checking the comment too, a parser left unchecked, the table reader left raw, and the writer back on the locale. Each is killed.

• Exception types:

  • Model.from_net and parse_net_file raise the ModelError and ValueError they document, where they raised UnicodeDecodeError;
  • build_model_from_parsed raises the RuntimeError it raises for any bad table line;
  • write_net no longer raises UnicodeEncodeError.

• A new skip is declared (_DECLARED_SKIPS), and the strict audit passes.

• Strict xfails and quarantines citing A .net file that is not UTF-8 fails with a bare UnicodeDecodeError: a Latin-1 species name Aµ() that loads as UTF-8 raises "can't decode byte 0xb5" from Model.from_net, and a refusal quoting such a byte loses its words #1653: none.

• In-flight PRs: antimony: load the main module, not the last one defined (#1508) #1642 and steady_state: say that steady_state_batch starts every entry from the initial conditions (#1521) #1643 also change _model.py, and antimony: load the main module, not the last one defined (#1508) #1642 the loading guide. Trial merges with all seven open PRs are clean.

• Claims are measured: each number here comes from a run above.

Not changed.

• The BNGL route. bngl_to_net reads a .bngl with Path.read_text(), in the locale's encoding, to key its cache, so a Latin-1 byte anywhere in a BNGL file, a comment included, fails there before BNG2.pl runs. About ten other readers and writers of BNGL, SED-ML and SBML text do the same. That is another file type and another reader, so it is left for an issue of its own.
• A UTF-8 byte-order mark at the start of a .net is refused as before, as a line outside any block, whose quote shows nothing visible.

The diff (abridged)

@@ include/bngsim/utf8.hpp (new) — Python's strict decoder, in C++
+inline std::size_t utf8_char_length(const std::string &s, std::size_t i) {
+    ...  // Table 3-7: C2..DF; E0 A0..BF; E1..EC, EE, EF; ED 80..9F; F0 90..BF; F1..F3; F4 80..8F
+}
+inline std::string shown_as_utf8(const std::string &s) {
+    ...  // each byte outside a character as \xhh, as backslashreplace shows it
+}

The second-byte ranges are where the overlong forms, the surrogates and the code points past U+10FFFF are excluded.

@@ src/net_file_loader.cpp — each block parser checks the text it reads
         if (ends_block(stripped, "reactions")) {
             got_end = true;
             break;
         }
+        require_utf8(stripped, "reaction");

The same line follows the end-of-block test in the parameters, species, functions and groups parsers. stripped is the line with its comment off, so a comment's bytes are not checked.

@@ src/net_file_loader.cpp — parse_net_file_structure's wrapper
     } catch (const std::exception &e) {
+        const std::string words = shown_as_utf8(e.what());
-        if (!holds_bare_cr(path))
-            throw;
+        if (!holds_bare_cr(path)) {
+            if (words == e.what())
+                throw;
+            throw std::runtime_error(words);
+        }
         ...
-        const char *p = e.what();
+        const char *p = words.c_str();

A refusal with no such byte is rethrown as it was. The bare-CR cut now runs over UTF-8 text.

@@ src/table_function.cpp — the reader's four quotes of the table
-            throw std::runtime_error("TableFunction '" + name + "': invalid data line: '" + line +
-                                     "'");
+            throw std::runtime_error("TableFunction '" + name + "': invalid data line: '" +
+                                     shown_as_utf8(line) + "'");

The header line and both header columns are shown the same way.

@@ python/bngsim/convert/_net_writer.py — write_net
-        Path(out_path).write_text(text)
+        Path(out_path).write_text(text, encoding="utf-8")

Tests

• python/tests/test_net_not_utf8.py (new, 28 cases):
  • Refused, with the line and the byte:
    • the issue's file, through Model.from_net and parse_net_file;
    • a byte in each of the five blocks;
    • three refusals of lines no block reads;
    • a bare-CR file's cut across runs of 10, 400 and 5,000 such bytes;
    • every edge of Table 3-7, against Python's decoder.
  • A table a .net reads:
    • a data row, either header column and a one-column header;
    • build_model_from_parsed reading the table again.
  • Read as before:
    • a UTF-8 name, run to the closed form;
    • a byte in a header comment, a comment line, a reaction's and a species' trailing comment, a block passed over and an action line;
    • a byte in a table file's comment.
  • write_net under the C locale with coercion off (ASCII on Linux and macOS, cp1252 on Windows): Aµ() and AΩ() round-trip.
• python/tests/conftest.py: the new skip's declaration.