[Montana]

Fixes #1515.

Important

NFsim seeds its generators from the seed's low 32 bits, so seed=2**32 + 12345 returned the seed=12345 trajectory bit for bit, while Result.seed and session.seed reported 4294979641. Two runs that read as independent were one, and no seed check could see it. Such a seed is now refused by name, and seeds up to 2**32 - 1 run as before.

The problem

On main (0b2055a9), with the issue's script (X(s~u) <-> X(s~p), 300 molecules), Xp at t = 1..5:

method	seed=12345	seed=2**32 + 12345	same?
nf	75, 116, 131, 155, 168	75, 116, 131, 155, 168	yes, and Result.seed = 4294979641
rm	64, 104, 140, 153, 171	72, 103, 132, 143, 161	no
ssa	72, 110, 153, 150, 167	78, 124, 148, 154, 177	no

NfsimSession did the same. Its initialize(seed=2**32) also replaced a live session with seed 0's and reported 4294967296.

The cause is in the vendored source. Both generators NFsim draws from mask the seed to its low word, as reference MT19937's init_genrand does:

• the System's NfsimRNG::seed (third_party/nfsim/src/NFutil/nfsim_rng.h) does state_[0] = s & 0xFFFFFFFFUL;
• the process-wide MTRand_int32::seed (NFutil/MTrand/mtrand.cpp) does the same.

SSA, PSA and RuleMonkey seed std::mt19937_64 with all 64 bits.

Separately, a seed below 0 or past 2**64 - 1 raised pybind11's TypeError: run(): incompatible function arguments on every method and session, though the docs said "any integer".

The fix

Important

Of the issue's three options I took the first: refuse an NFsim seed past 232 - 1**. Every seed up to 2**32 - 1 keeps its trajectory bit for bit. numpy's legacy MT19937, the same algorithm, refuses such a seed. NFsim's own command line reads -seed as an int, so no BNG2.pl parity case needs a larger one. Widening the seed would need a change to vendored code and to the process-wide MTRand that #1438 proposes retiring.

src/nfsim_simulator.cpp:

• $\color{#8250df}{\textsf{Refuse what NFsim cannot hold.}}$ require_nfsim_seed raises std::invalid_argument (a ValueError in Python), naming the seed it would have run as: "NFsim seeds its generators from 32 bits, so seed 4294979641 would run as seed 12345: an NFsim seed must be from 0 to 4294967295 (2**32 - 1)".
• $\color{#8250df}{\textsf{Check before anything changes.}}$ It is called first in NfsimSimulator::run and NfsimSimulator::initialize, so the XML is not read for a seed that will be refused, and initialize does not drop a live session for one. Every NFsim entry point goes through these two:
  • Simulator.run with nf, nfsim or nf_reject;
  • NfsimSession.initialize;
  • the private _bngsim_core.NfsimSimulator bindings.

python/bngsim/_seed.py:

• $\color{#8250df}{\textsf{Name the range.}}$ _check_seed refuses a seed outside 0 to 264 - 1 with "seed must be an integer from 0 to 264 - 1, got -1". _resolve_seed calls it, which covers every stochastic method, run_batch, run_replicates, parameter_scan and both sessions. The ODE path, which reads an explicit seed for equal-priority event ties (GH fix(sens): make the default sensitivity column set independent coordinates (#203) #242), calls it too.

Docs:

• docs/quickstart.md's seed contract gives the range, and says NFsim takes 0 to 2**32 - 1.
• The Result.seed, Simulator.run, NfsimSession.initialize and RuleMonkeySession.initialize docstrings say the same.
• Simulator.run's docstring said the seed is "Ignored for method="ode"". An ODE run with events uses it for tie-breaking (opts.event_seed), so that line now says so.
• The _seed.py docstring said NFsim narrows a seed only on Windows; it narrows it everywhere.

How this was checked

1. Reproduced on main with the issue's script: the table above, and the session.
2. Fixed:
   • nf refuses 2**32, 2**32 + 12345, 2**63 and 2**64 - 1, each naming its low word.
   • 2**32 - 1 runs and is its own trajectory, not seed 0's.
   • A refused initialize leaves the session at seed 7, and its next simulate equals a fresh seed-7 session's.
3. Nothing below 232 moves:** every NFsim fixture under tests/data/nfsim/, 85 models, at seeds 0, 1, 12345, 231 and 232 - 1: 425 of 425 runs give the same Result.seed and the same trajectory digest on main and here.
   • 280 of the runs complete; the other 145 error the same way on both sides (fixtures written for other setups).
   • The trajectories at seeds 0, 12345 and 2**32 - 1 of simple_system.xml, and of a seed-7 session, are bit-identical too.
4. The range, on every method: on main, ode, ssa, nf, rm and both sessions raised TypeError for -1 and 2**64; here each raises the ValueError above. PSA takes the same path, pinned in the tests.
5. New tests fail on main for the right reason: 30 cases in two files.
   • test_nfsim_seed_width.py, 9 cases: 7 fail on main (the four seeds, the message, the session, the live session); 2 controls pass (2**32 - 1 is its own run; a 32-bit seed reproduces).
   • test_seed_range.py, 21 cases: 12 fail on main (ode, ssa and psa runs, run_replicates, and both sessions, each at -1 and 2**64); 9 controls pass (SSA and PSA keep all 64 bits at 0, 232, 263 and 2**64 - 1, and RuleMonkey runs 2**32 + 12345).
6. Mutation tests: 7 mutants, all killed, rerun after clang-format:
   • the guard removed from run() (5 failing);
   • the guard removed from initialize() (2);
   • initialize() checking after it drops the session (1);
   • the bound off by one (1, the 2**32 - 1 control);
   • the message naming the high word (5);
   • the Python range check removed (12);
   • the ODE path skipping it (2).
7. C++ tests: ctest, 8 of 8 pass.
8. Special legs over the 247 test files that use a seed or a stochastic method, on the final tree:
   • BNGSIM_SKIP_AUDIT=strict: 8,683 passed, 41 skipped, 1 xfailed;
   • forced cc (BNGSIM_CODEGEN_THRESHOLD=1): 8,676 passed, 48 skipped, 1 xfailed;
   • forced MIR: 8,676 passed, 48 skipped, 1 xfailed.
9. Full Python suite, macOS arm64, no corpus (as CI runs it), on 0b2055a9, on the final tree: 19,393 passed, 140 skipped, 1 xfailed, 0 failed, so 19,534 collected, main's 19,504 plus these 30.
   • Collected floor: 83 above PYTEST_COLLECTED_FLOOR (19,451), inside the 500 slack.
   • Skips: 140, under PYTEST_SKIPPED_CEILING (252). The new tests add no skip on this build, which has NFsim and RuleMonkey.
10. Docker, Linux x86-64 (python:3.12, --platform linux/amd64, gcc 14, built from the final tree with the [antimony,hdf5] extras and scipy):

• The two new files: 30 passed.
• Full suite: 18,829 passed, 537 skipped, 1 xfailed, 4 failed. The 4 are this container's standing failures (test_codegen_compile_reaping, test_codegen_toolchain_verdict, test_run_tests_sh, test_vendor_script_git_cwd), which fail the same way on main in it and on simulator: leave assignment-rule species at the value a run reports (#1510) #1637's, net: refuse a .net file from which no block is read (#1511) #1638's and net: refuse a species index that is not its line (#1512) #1639's runs; none touches a seed.

11. pre-commit --all-files passes (13 hooks, clang-format, ruff and mypy among them). ci/changelog.py check passes, and local sphinx-build -W exits 0.

Which CI legs this PR runs

The MIR and Windows legs run only on the files in their paths: filters. This PR changes src/nfsim_simulator.cpp, _nfsim_session.py, _simulator.py, _seed.py, _result.py, _rulemonkey_session.py, two test files, a changelog fragment and a docs page.

leg	runs here?	what covers it
python-tests.yml (Ubuntu, macOS)	yes	the whole suite
native-tests.yml	yes (src/**)	ctest, 8 of 8 locally
windows-nfsim.yml (MSVC)	yes, through src/nfsim_simulator.cpp and _nfsim_session.py	it runs python/tests/test_nfsim*.py, so test_nfsim_seed_width.py runs under MSVC
windows-tail.yml, windows-klu.yml (MSVC)	yes, through _simulator.py	their own lists, test_seed_semantics.py among windows-tail's; test_seed_range.py is not on them
mir.yml	yes, through _simulator.py	its own list; forced MIR locally on the 247 files (step 8)

Checked against past review findings

• Sibling sites:
  • Both NFsim entry points are guarded in C++, which also covers the private bindings.
  • The range check sits in _resolve_seed, which every stochastic method, batch, scan and session uses, and in the ODE path.
  • run_until with nf raises NotImplementedError ("Interactive simulation helpers are not supported for XML-backed network-free backends"), so it does not reach NFsim. The issue thought it reached NFsim through run; it doesn't.
• Exception types:
  • A seed past 2**32 - 1 under NFsim now raises ValueError where it ran.
  • A seed below 0 or past 2**64 - 1 now raises ValueError where pybind11 raised TypeError. No test or caller in the repository catches that TypeError.
• Stale docstrings: the "Ignored for method="ode"" line and the "32 bits on Windows" line are corrected, beside the ones the issue names.
• The live session: a refused re-initialize was the one path that could have lost state. It is pinned against a fresh session.
• Strict xfails and quarantines citing NFsim uses only the low 32 bits of seed=: seed=4294979641 (2**32 + 12345) returns the seed-12345 trajectory while Result.seed reports 4294979641 #1515: none.
• In-flight PRs: none of the open PRs touches these files. Trial merges with net: read a Hill law with 0 < h < 1 as 0 at zero substrate, as run_network does (#1497) #1629, cvode: take an output at the start of a plain run as reached, as the event-capable loop does (#1501) #1633, result: write the species a Result reports to .cdat, in the core's layout (#1505) #1635, simulator: leave assignment-rule species at the value a run reports (#1510) #1637, net: refuse a .net file from which no block is read (#1511) #1638 and net: refuse a species index that is not its line (#1512) #1639 are each clean.
• Count bounds, both directions: step 9 for the Python suite. windows-nfsim.yml runs every test_nfsim*.py, so the 9 new cases take it from 386 passed to 395, past its floor's slack (378 + 16). ci/test_floors.env raises NFSIM_PASSED_FLOOR to 387, 8 below the count, as its note asks (afd709c).
• Claims are measured: each number here comes from a run above.

The diff (abridged)

@@ src/nfsim_simulator.cpp — the guard, and its two call sites
+static void require_nfsim_seed(uint64_t seed) {
+    constexpr uint64_t kMaxSeed = 0xFFFFFFFFull;
+    if (seed > kMaxSeed) {
+        throw std::invalid_argument("NFsim seeds its generators from 32 bits, so seed " +
+                                    std::to_string(seed) + " would run as seed " +
+                                    std::to_string(seed & kMaxSeed) +
+                                    ": an NFsim seed must be from 0 to 4294967295 (2**32 - 1)");
+    }
+}
+
 Result NfsimSimulator::run(const TimeSpec &times, uint64_t seed, double timeout_seconds) {
+    require_nfsim_seed(seed);
 ...
 void NfsimSimulator::initialize(uint64_t seed) {
+    require_nfsim_seed(seed);
     // Destroy any existing session

std::invalid_argument reaches Python as ValueError. It is thrown before prepare_system, whose catch would have wrapped it as "NFsim setup failed".

@@ python/bngsim/_seed.py — the range
+def _check_seed(seed: int) -> int:
+    value = int(seed)
+    if not 0 <= value <= _MAX_SEED:
+        raise ValueError(f"seed must be an integer from 0 to 2**64 - 1, got {value}")
+    return value
 ...
     if seed is None:
         return secrets.randbits(31)
-    return int(seed)
+    return _check_seed(seed)

@@ python/bngsim/_simulator.py — the ODE path's explicit seed
-            used_seed = _DEFAULT_EVENT_SEED if seed is None else int(seed)
+            used_seed = _DEFAULT_EVENT_SEED if seed is None else _check_seed(seed)

Tests

• python/tests/test_nfsim_seed_width.py (new, 9 cases; runs under MSVC through windows-nfsim.yml):
  • four seeds past 2**32 - 1, each naming its low word;
  • the message naming the range;
  • a session refusing one;
  • a refused re-initialize leaving the live session;
  • controls: 2**32 - 1 is its own run; a 32-bit seed reproduces.
• python/tests/test_seed_range.py (new, 21 cases):
  • -1 and 2**64 refused on ode, ssa and psa runs, run_replicates, and both sessions;
  • controls: SSA and PSA keep all 64 bits, and RuleMonkey runs 2**32 + 12345.

[elyfmiller]

Montana, thanks. The fix reproduces here. On the PR head (afd709c, arm64 with MIR), the two new files' 30 cases pass; on its base (0b2055a), 19 fail and the 11 controls pass, as the body says, and v0.17.1 fails the same 19. The issue's script gives the issue's table on the base, and on the head seed=2**32 + 12345 under nf raises the ValueError naming seed 12345. The branch merges cleanly with today's main (80b15ae), and onto it with each of #1641, #1642, #1643 and #1645; only #1643 shares a file (_simulator.py).

What else we checked:

• Every NFsim fixture under tests/data/nfsim/ (85), at seeds 0, 1, 12345, 231 and 232 - 1, base against head: 425 of 425 runs give the same Result.seed and trajectory digest. 280 run and 145 fail the same way on both, as the body says.
• Other paths: nf and rm runs refuse -1 with the range, and nf refuses np.uint64(2**32) as it refuses 2**32. Under SSA a seed of 2**64 - 1 survives Result.save and load and to_xarray, and run_replicates from it reports it on row 0.
• The full Python suite on today's main with our commit merged: 19,154 passed, 402 skipped, 1 xfailed and 1 failed. The one failure, test_run_tests_sh, wants optional extras this machine lacks (jax, diffrax, roadrunner, vivarium-core), and fails the same way on v0.17.1. Our commit changes only the two test files. Your CI on afd709c, including the NFsim leg on Windows, is green.

We've pushed one commit on top of yours, 6fd19f3, for items 1 and 2:

1. The 64-bit controls checked only Result.seed. That is the check NFsim uses only the low 32 bits of seed=: seed=4294979641 (2**32 + 12345) returns the seed-12345 trajectory while Result.seed reports 4294979641 #1515 showed cannot see a masked seed: NFsim reported 2**32 + 12345 and ran 12345. A backend masking its seed to 32 bits passed test_a_seed_in_range_runs_and_is_reported and test_rulemonkey_keeps_all_64_bits.
   • The new test: test_a_seed_past_32_bits_is_a_run_of_its_own runs SSA and PSA at 12345 and at 2**32 + 12345 and checks that the trajectories differ (A_tot 95 against 87 at t = 1). The RuleMonkey control now does the same (X_p_total 296 against 262).
   • Mutants: masking the seed passed to SSA, PSA or RuleMonkey fails its control. Each passed before.
2. The refused re-initialize was checked at t = 0. A session put back to its start with its own seed, rather than left where it was, passed.
   • The change: the session runs to t = 5, is refused, and runs on to 10. That second half must equal the one from a seed-7 session that was never refused. The two sessions run one after the other, since each initialize reseeds NFsim's process-wide generator.
   • Mutant: a session that re-initializes with its own seed on a refusal fails it. It passed before.

Not changed:

3. Under method="ode", run_batch and parameter_scan don't check the range. run(seed=-1) is refused, but run_batch(..., seed=-1) and parameter_scan(..., seed=2**64) run, since those paths read no seed. Nothing runs wrong, so it's a note, but the changelog's "on every method and session" is wider than the code.
4. A negative seed under NFsim is told the 64-bit range. NfsimSession.initialize(seed=-1) gets "from 0 to 264 - 1", and a caller who then tries 263 is refused again, with the 32-bit message.
5. RuleMonkeySession.initialize has no Raises entry for the ValueError, which NfsimSession.initialize lists.
6. eval_spec: carry a spec's seed through its JSON, so a stochastic spec runs one trajectory on every worker (#1517) #1641 checks the same range in EvaluationSpec, and also refuses a bool or a non-integer seed, which _check_seed truncates (run(seed=1.7) runs seed 1, as on main). Whichever merges second could share one check.