Skip to content

rr_parity: give RoadRunner 100000 steps per output interval - #1676

Merged
wshlavacek merged 2 commits into
mainfrom
fix/rr-ode-step-budget
Nov 12, 2026
Merged

wshlavacek merged 2 commits into
mainfrom
fix/rr-ode-step-budget

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

rr_ode left RoadRunner its default maximum_num_steps of 20000, a hard ceiling per output interval. bngsim's max_steps is also 20000, but it is a batch size: bngsim calls CVode again while the time advances, and raises only when a batch makes no progress (retry_while_advancing, #54). So at the same number, RoadRunner gave up on runs bngsim completed. #1654 found BIOMD0000000375 REFERENCE_FAILED for that reason alone. This PR gives RoadRunner 100000 steps per output interval (_rr_common.RR_MAX_STEPS).

What it changes

BIOMD0000000375 goes from REFERENCE_FAILED to PASS. At its TOL_OVERRIDES tolerance (rtol 1e-11, atol 1e-16), one 800-unit output interval needs about 21000 steps: 20500 fails, 21000 passes. With the budget, RoadRunner completes and passes against bngsim at the same tolerance (err/movement 3.4e-3) and at 1e-12 (2.8e-3). bngsim converges there too (against its own 1e-12 run, 5.8e-4).

No other row changes verdict. Measured with RoadRunner 2.11.0 on macOS arm64, on every row that ran out of steps at 20000 in the 0.15.1 paper reports or in the nightly baseline, at four budgets:

row 20000 40000 / 100000 / 1000000 RoadRunner stops at
BIOMD0000000375 REFERENCE_FAILED PASS completes
BIOMD0000000338 REFERENCE_FAILED REFERENCE_FAILED t = 0.0971 at every budget
MODEL1112050000 REFERENCE_FAILED REFERENCE_FAILED t = 10.64 at every budget
MODEL1112050001 REFERENCE_FAILED (adjudicated uncoverable) the same; the error becomes CV_CONV_FAILURE t = 3.36e-7, the overflow its adjudication describes
MODEL2001310001 BAD_TEST the same; the error becomes CV_CONV_FAILURE t = 43.59, where bngsim stalls too
MODEL2003050002 BAD_TEST BAD_TEST t = 0.0586
MODEL0995500644 BAD_TEST BAD_TEST an error-test failure at t = 0.377 on macOS

The other eight rows give the same verdict at every budget: six PASS, including 339's known artifact, and two TIMEOUT. The rows that stay are stalls, not slow runs, so a larger budget only makes them fail later in wall time: 338 takes about 1 s at 100000 and 13 s at 1000000. 100000 is also the budget the AMICI sensitivity harness gives both engines (#339).

Table 17

Only libRoadRunner's column moves: Failed −1, Validated +1, for 375. BNGsim's column is unchanged, because COPASI already passes 375 in the 0.15.1 reports.

The nightly

A run that completes comes out bit-identical under a larger budget: the review checked all 1323 jobs, 20000 against 100000. So the budget stays out of the archive's identity, where it would stale every entry at once. An entry where RoadRunner ran out of steps does depend on it, and CVODE names the budget in that message (mxstep (20000)). archive_load now treats such an entry, recorded under a budget other than RR_MAX_STEPS, as archive_stale.

Until the next recording:

  • the Sunday live run passes 375 (REFERENCE_FAILED → PASS, not an alert);
  • the replays show the rows the current archive recorded as out of steps (338, 375 and MODEL1112050001; on Linux also MODEL0995500644, MODEL2001310001 and MODEL2003050002) as REFERENCE_FAILED archive_stale. None was good, so none alerts;
  • the Monday replay after a live night reports 375 as "archive gap after a live night, as the committed baseline records" (nightly: a replay after the live Sunday run alerts on rows the archive can't replay (rr_ssa MODEL2003200001) #1599), not as a regression. I checked this with the nightly's own extract and diff, on a live run passing 375 followed by a replay of a 20000-step refusal.

A recording settles it all; the committed baseline then re-records 375 as PASS.

RoadRunner with the new budget has not run on Linux: pull requests replay the archive. The review ran x86_64 RoadRunner 2.11.0 under Rosetta, and it fails on the same rows, with 375 needing the same steps (it fails at 20961 and completes at 20976). MODEL0995500644 there runs out of steps at t = 0.4029, at the same t at every budget, so it stays BAD_TEST.

Also

  • 375's reason and its evidence test now say the row passes, and keep why it was REFERENCE_FAILED. The test fails with the budget set back to 20000.
  • MODEL1112050001's reason names both errors RoadRunner stops with at t ~ 3.36e-7.
  • rr_ode's timing config records max_steps, so an archived entry says which budget it ran under.
  • ode_jobs.json re-baked with build_ode_jobs.py --mirror data/sedml_source/final. Only those two rows change, and _meta is untouched.
  • The rr_parity README gains a "Step budget" section.
  • ci/test_floors.env: parity floor 1381 → 1382, for one new test. Counted as CI runs: 1392 collected, 83 skipped.

Tests

  • test_rr_reference_archive.py, one new test: an out-of-steps entry under another budget is archive_stale, and one under RR_MAX_STEPS replays. It fails without the rule.
  • test_rr_known_artifact_evidence.py: 36 passed with the corpus.
  • The parity suite with the corpus: 1346 passed, 46 skipped.
  • A live rr_run.py on the 15 rows above gives the table's 100000 column.
  • The pinned pre-commit hooks pass.

Review

One round. No HIGH findings, and every figure in the first commit reproduced. Check 3 asked whether 375's PASS checks anything: the engines agree at rtol 1e-13 (6.1e-5), and RoadRunner at 1e-11 is 3.2e-3 from its own 1e-13 run, so both sides are converged. Fixed:

  1. MEDIUM: a false nightly alert every Monday. Sunday's live run would pass 375, and Monday's replay of the recorded CV_TOO_MUCH_WORK diffed as a regression. Fixed by the archive_stale rule above.
  2. LOW: the evidence test pinned RoadRunner failing at 20000 steps. That pins RoadRunner, not the harness, and the test fails with the old budget anyway. Dropped.
  3. LOW: the comment's list of stalls left out MODEL0995500644, and said "a budget changes only an entry that raised", which holds only for a larger one. Both corrected, and the budget is now in the timing config.
  4. LOW: not run on Linux. Covered under Rosetta, as above. The Sunday live night is the first Linux run.

rr_ode left RoadRunner its default maximum_num_steps of 20000, a hard
ceiling per output interval. bngsim's max_steps is also 20000, but it is a
batch size: bngsim calls CVode again while the time advances and raises
only when a batch makes no progress (retry_while_advancing). So at the same
number RoadRunner gave up on runs bngsim completed.

BIOMD0000000375 at its TOL_OVERRIDES tolerance (1e-11/1e-16) needs about
21000 steps for one 800-unit output interval (20500 fails, 21000 passes),
and was REFERENCE_FAILED only for that. With the budget it completes and
passes against bngsim at the same tolerance (err/movement 3.4e-3) and at
1e-12 (2.8e-3), so the row is a PASS.

Measured with RoadRunner 2.11.0 on macOS arm64, on every row that ran out
of steps at 20000 in the 0.15.1 paper reports or the nightly baseline, at
20000, 40000, 100000 and 1000000. No other row changes verdict. The rest
stall: RoadRunner stops at the same time at 1000000 steps as at 20000
(BIOMD0000000338 at t = 0.0971, MODEL1112050000 at 10.64, MODEL1112050001
at 3.36e-7, MODEL2001310001 at 43.59, MODEL2003050002 at 0.0586). A larger
budget makes them fail later in wall time: 338 takes about 1 s at 100000,
13 s at 1000000. 100000 is the AMICI sensitivity harness's budget (#339).

375's reason and evidence test now say the row passes, and keep why it was
REFERENCE_FAILED. MODEL1112050001's reason names both errors RoadRunner
stops with at t~3.36e-7 (CV_CONV_FAILURE with the new budget). ode_jobs.json
re-baked; only those two rows change. The reference archive's identity
does not carry the budget, so the nightly's replays keep 375 at
REFERENCE_FAILED until the next recording, and its Sunday live run passes
it.
…tale

Review of the budget change: the nightly replays a recorded archive six
nights a week and runs RoadRunner live on Sundays. With rr_ode's new budget
the live night passes BIOMD0000000375, and the next replay brought back the
recorded CV_TOO_MUCH_WORK, which the night-to-night diff alerted on as a
regression every Monday until the archive is re-recorded.

The budget is not part of an entry's identity, since a completed run comes
out bit-identical under a larger budget (all 1323 jobs, 20000 against
100000); putting it there would stale every entry at once. An entry where
RoadRunner ran out of steps does depend on it, and CVODE writes the budget
into that message. So archive_load treats "mxstep (N)" with N other than
RR_MAX_STEPS as archive_stale. A replay then reports such a row as an
archive gap, which after a live night the diff reports instead of alerting
(#1599): checked with the nightly's own extract and diff on a Sunday live
PASS followed by a Monday replay of a 20000-step refusal for 375.

Also from the review:
- rr_ode's timing config records max_steps, so an archived entry says which
  budget it ran under.
- The evidence test no longer pins RoadRunner failing at 20000 steps: that
  pins RoadRunner, and the test fails with the old budget anyway.
- The comment's list of stalls gains MODEL0995500644 (t = 0.4029 at every
  budget on x86_64; on arm64 it fails its error test first), and states
  that completed runs are bit-identical.
- ci/test_floors.env: parity floor 1381 -> 1382 for the new archive test.
@wshlavacek
wshlavacek merged commit 7fcd706 into main Nov 12, 2026
35 checks passed
@wshlavacek
wshlavacek deleted the fix/rr-ode-step-budget branch November 12, 2026 21:33
wshlavacek added a commit that referenced this pull request Nov 13, 2026
nightly: pin reference-data-v7, recorded with #1676's RoadRunner step budget
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant