Repository navigation
rr_parity: give RoadRunner 100000 steps per output interval - #1676
Merged
Merged
Conversation
rr_ode left RoadRunner its default maximum_num_steps of 20000, a hard ceiling per output interval. bngsim's max_steps is also 20000, but it is a batch size: bngsim calls CVode again while the time advances and raises only when a batch makes no progress (retry_while_advancing). So at the same number RoadRunner gave up on runs bngsim completed. BIOMD0000000375 at its TOL_OVERRIDES tolerance (1e-11/1e-16) needs about 21000 steps for one 800-unit output interval (20500 fails, 21000 passes), and was REFERENCE_FAILED only for that. With the budget it completes and passes against bngsim at the same tolerance (err/movement 3.4e-3) and at 1e-12 (2.8e-3), so the row is a PASS. Measured with RoadRunner 2.11.0 on macOS arm64, on every row that ran out of steps at 20000 in the 0.15.1 paper reports or the nightly baseline, at 20000, 40000, 100000 and 1000000. No other row changes verdict. The rest stall: RoadRunner stops at the same time at 1000000 steps as at 20000 (BIOMD0000000338 at t = 0.0971, MODEL1112050000 at 10.64, MODEL1112050001 at 3.36e-7, MODEL2001310001 at 43.59, MODEL2003050002 at 0.0586). A larger budget makes them fail later in wall time: 338 takes about 1 s at 100000, 13 s at 1000000. 100000 is the AMICI sensitivity harness's budget (#339). 375's reason and evidence test now say the row passes, and keep why it was REFERENCE_FAILED. MODEL1112050001's reason names both errors RoadRunner stops with at t~3.36e-7 (CV_CONV_FAILURE with the new budget). ode_jobs.json re-baked; only those two rows change. The reference archive's identity does not carry the budget, so the nightly's replays keep 375 at REFERENCE_FAILED until the next recording, and its Sunday live run passes it.
…tale Review of the budget change: the nightly replays a recorded archive six nights a week and runs RoadRunner live on Sundays. With rr_ode's new budget the live night passes BIOMD0000000375, and the next replay brought back the recorded CV_TOO_MUCH_WORK, which the night-to-night diff alerted on as a regression every Monday until the archive is re-recorded. The budget is not part of an entry's identity, since a completed run comes out bit-identical under a larger budget (all 1323 jobs, 20000 against 100000); putting it there would stale every entry at once. An entry where RoadRunner ran out of steps does depend on it, and CVODE writes the budget into that message. So archive_load treats "mxstep (N)" with N other than RR_MAX_STEPS as archive_stale. A replay then reports such a row as an archive gap, which after a live night the diff reports instead of alerting (#1599): checked with the nightly's own extract and diff on a Sunday live PASS followed by a Monday replay of a 20000-step refusal for 375. Also from the review: - rr_ode's timing config records max_steps, so an archived entry says which budget it ran under. - The evidence test no longer pins RoadRunner failing at 20000 steps: that pins RoadRunner, and the test fails with the old budget anyway. - The comment's list of stalls gains MODEL0995500644 (t = 0.4029 at every budget on x86_64; on arm64 it fails its error test first), and states that completed runs are bit-identical. - ci/test_floors.env: parity floor 1381 -> 1382 for the new archive test.
wshlavacek
added a commit
that referenced
this pull request
Nov 13, 2026
nightly: pin reference-data-v7, recorded with #1676's RoadRunner step budget
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rr_odeleft RoadRunner its defaultmaximum_num_stepsof 20000, a hard ceiling per output interval. bngsim'smax_stepsis also 20000, but it is a batch size: bngsim calls CVode again while the time advances, and raises only when a batch makes no progress (retry_while_advancing, #54). So at the same number, RoadRunner gave up on runs bngsim completed. #1654 found BIOMD0000000375 REFERENCE_FAILED for that reason alone. This PR gives RoadRunner 100000 steps per output interval (_rr_common.RR_MAX_STEPS).What it changes
BIOMD0000000375 goes from REFERENCE_FAILED to PASS. At its TOL_OVERRIDES tolerance (rtol 1e-11, atol 1e-16), one 800-unit output interval needs about 21000 steps: 20500 fails, 21000 passes. With the budget, RoadRunner completes and passes against bngsim at the same tolerance (err/movement 3.4e-3) and at 1e-12 (2.8e-3). bngsim converges there too (against its own 1e-12 run, 5.8e-4).
No other row changes verdict. Measured with RoadRunner 2.11.0 on macOS arm64, on every row that ran out of steps at 20000 in the 0.15.1 paper reports or in the nightly baseline, at four budgets:
The other eight rows give the same verdict at every budget: six PASS, including 339's known artifact, and two TIMEOUT. The rows that stay are stalls, not slow runs, so a larger budget only makes them fail later in wall time: 338 takes about 1 s at 100000 and 13 s at 1000000. 100000 is also the budget the AMICI sensitivity harness gives both engines (#339).
Table 17
Only libRoadRunner's column moves: Failed −1, Validated +1, for 375. BNGsim's column is unchanged, because COPASI already passes 375 in the 0.15.1 reports.
The nightly
A run that completes comes out bit-identical under a larger budget: the review checked all 1323 jobs, 20000 against 100000. So the budget stays out of the archive's identity, where it would stale every entry at once. An entry where RoadRunner ran out of steps does depend on it, and CVODE names the budget in that message (
mxstep (20000)).archive_loadnow treats such an entry, recorded under a budget other thanRR_MAX_STEPS, asarchive_stale.Until the next recording:
archive_stale. None was good, so none alerts;extractanddiff, on a live run passing 375 followed by a replay of a 20000-step refusal.A recording settles it all; the committed baseline then re-records 375 as PASS.
RoadRunner with the new budget has not run on Linux: pull requests replay the archive. The review ran x86_64 RoadRunner 2.11.0 under Rosetta, and it fails on the same rows, with 375 needing the same steps (it fails at 20961 and completes at 20976). MODEL0995500644 there runs out of steps at t = 0.4029, at the same t at every budget, so it stays BAD_TEST.
Also
rr_ode's timing config recordsmax_steps, so an archived entry says which budget it ran under.ode_jobs.jsonre-baked withbuild_ode_jobs.py --mirror data/sedml_source/final. Only those two rows change, and_metais untouched.ci/test_floors.env: parity floor 1381 → 1382, for one new test. Counted as CI runs: 1392 collected, 83 skipped.Tests
test_rr_reference_archive.py, one new test: an out-of-steps entry under another budget isarchive_stale, and one underRR_MAX_STEPSreplays. It fails without the rule.test_rr_known_artifact_evidence.py: 36 passed with the corpus.rr_run.pyon the 15 rows above gives the table's 100000 column.Review
One round. No HIGH findings, and every figure in the first commit reproduced. Check 3 asked whether 375's PASS checks anything: the engines agree at rtol 1e-13 (6.1e-5), and RoadRunner at 1e-11 is 3.2e-3 from its own 1e-13 run, so both sides are converged. Fixed:
archive_stalerule above.