Repository navigation
Conversation
Copy the 30-day forecast tables into forecast-archive.db after startup. The file stays off the boot and planning path, keeps full detail for 30 days, then one run per hour for two years, and stops before a 512 MiB cap or a full data volume. Health reports it without changing box status. Full backups store a consistent copy. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: daed653b55
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| query := `SELECT source, available_at_ms, start_ms, end_ms, ghi, cloud, temp, direct_pv, est_pv FROM fa_weather | ||
| WHERE available_at_ms IN (` + placeholders(len(ids)) + `) AND start_ms < ? AND end_ms > ? ORDER BY start_ms, source` |
There was a problem hiding this comment.
Preserve exact weather-row membership for each archived run
When a forecast uses a mixture of fetch timestamps—for example, when a partial provider refresh leaves an older cached slot—this query retrieves every overlapping row from both fetches, not just the rows captured by that run. Rows from the older fetch that were replaced in the issue are consequently added back alongside their replacements. I reproduced this with two valid archived issues: ArchiveIssues succeeds, but LoadIssue then fails validation with weather unavailable at forecast origin because the reconstructed weather intervals overlap. Store each run's exact weather-row references rather than only its availability timestamps.
Useful? React with 👍 / 👎.
| rows, err := a.db.QueryContext(ctx, `SELECT run_id, origin_ms, timezone FROM fa_runs | ||
| WHERE thinned=0 AND origin_ms<? ORDER BY origin_ms, run_id`, cutoff) |
There was a problem hiding this comment.
Include previously retained winners when thinning cutoff batches
When daily maintenance runs partway through an hour, the 30-day cutoff splits that hour across maintenance passes. The first pass marks its retained run as thinned; the next pass excludes that winner here and independently retains another run from the same UTC hour. I reproduced this with runs at 12:00, 12:15, 12:30 and 12:45 and successive daily cutoffs at 12:02: two runs remain permanently instead of one. This also retains an additional model-state snapshot for the same local day. Compare newly eligible runs against previously retained winners, or process only complete expired hours.
Useful? React with 👍 / 👎.
The archive linked a run only to the fetch times it used and, on load, returned every row of those fetches that overlapped the series span. After a partial weather refresh one issue holds rows from two fetches, so another run's row for the same slot came back too and the issue failed validation. Rows outside the span were dropped. fa_run_weather now holds one row per run and fetch, keyed by run, source and fetch time, with the run's slot starts packed as uvarint gaps. LoadIssue reads exactly those rows, all of them, in start order. The fa_runs weather_avail column is gone. The fa-v1 schema changes in place; no box has the file yet. Garbage collection probed fa_run_weather by fetch time with no index, so every candidate row scanned all links. A new index on (source, available_at_ms) serves the weather probe, and the model-state probe uses fa_run_states_ref. A test checks both query plans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
…floor Each thinning pass split the hour and the local day at the cutoff, so the rest of a split hour kept a second run and a split day kept a second set of model states. In half-hour zones every hour at a day boundary kept two runs. Below the disk floor the deletes refused to run, so nothing was thinned, expired or pruned, and the cap waited for the daily pass. Thinning now works on complete UTC hours. It ranks every run of the hour, keeps a thinned survivor if there is one, and so drops a run copied late into a thinned hour. Model states stay on the first kept run of each complete local day that has them. A partial index finds unthinned runs. Deletes no longer check the floor. Below it, copying stays paused but maintenance runs, and the cap target drops by the missing free space, so the oldest days outside the newest 30 give that space back. Every Sync enforces the cap, and incremental vacuum ends with a checkpoint so the file shrinks at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
…l damage The worker held the store's archive lock while it opened the file. A quick_check of a large file on an SD card made /api/health, the storage inventory, backups and shutdown wait behind it. The open now runs without that lock. A second lock keeps two opens apart, and a store that closes in the meantime gets no handle. Any quick_check error moved a good file aside, so a passing I/O, memory or full-disk error cost the whole archive. Now only a check that finds problems, or a corruption code, does that. Other errors report unavailable and the worker tries again next hour. Each heal removes older quarantined copies, so at most one stays. Backups still leave them out. Damage found while copying waited for a restart. Sync now closes the handle when the archive returns a corruption code, so the next hourly run checks and heals the file. A corruption code read from state.db does not count. The worker gave the open 30 seconds. A full quick_check can take longer on a slow card, and then every hourly try fails the same way. The open is off the boot path and nothing waits on it, so it now runs until it ends or shutdown cancels the worker's context. Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The copier read strictly after its cursor. A shadow issue saved just after its primary, with the same issued time and an id that sorts first, was skipped for good. Observations were read in start order, so a quarter released after a later one was skipped too. A cursor set while the clock ran ahead stopped all copying until that time came. Observations now follow available_at_ms, the order the tracker writes them in. Each copy reads again from an hour before its cursor and never past the clock plus the five minutes state.db accepts, so a row stamped by a wrong clock cannot pull the cursor ahead. A cursor already past that bound resets, and the copy reads the 30-day tables again. A rescan stays cheap: the copier lists issue keys without payloads and loads only the runs the archive does not hold. Rows read again are ignored on insert. The cursor and last_copy_ms move only when the copy gets past the old cursor or stores a run. The fa-v1 cursor changes in place; an old cursor reads as zero and starts one full rescan. Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three guards in the archive had no test that would fail without them. The daily thinning tests ran 2 minutes past the hour, when the open hour held only its run on the hour, so a pass that thinned a partial hour still passed. They now run at 18:40 UTC: the open hour holds runs a bad pass would delete, and a Kolkata day has just ended inside an open hour. A plan test now checks that the thinning queries read fa_runs through fa_runs_unthinned and fa_runs_origin. Another test deletes a linked weather row and expects LoadIssue to fail. The cleanup removes code nothing used. The delete helpers lose their one-value table and column arguments, forecastLongDay loses its unread key, NewestForecastIssueMS goes, and the observation reader returns a slice with no nil check or limit clamp, as the issue reader does. incrementalVacuum checkpoints only at the end: the vacuum sees pages freed in the WAL, and only the last checkpoint shrinks the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
- Skip an issue or observation the archive cannot store, with a warning, instead of stopping the copy at that row until state.db prunes it. - After a corruption code, move the file aside on the next open without asking quick_check again, and wait until the last read on the dropped handle has ended, so its close cannot delete the new file's WAL. - Health shows a short reason instead of raw errors with host paths, and an archive heal no longer replaces the boot storage event. - Log model states that state.db pruned before the copy. - Name forecast-archive.db in the operations and backup docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
… 1 GiB The Energyplan state has a fixed shape (56 days of 72 load buckets and 216 PV cells), about 450 KiB of JSON and 75-100 KiB gzip, and the worker writes a new one about every quarter. Kept on every run, it filled about 250 MiB of the newest 30 days, so 512 MiB held only four to six months. Inside the newest 30 days a run now keeps its model states only when it is on the hour, or when it is the first run copied into an hour that has none. Thinning already keeps the run on the hour, and the states between can be rebuilt from the archived observations. With the cap at 1 GiB, two years fit by the same estimate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
In an hour with no run on the hour, the first run copied holds the model states. A run issued earlier but copied later used to win the hour, so the holder was deleted and the day could lose its states. Hour thinning now ranks a run with states first. The plan test covers the new hour query and checks fa_run_states is never scanned. The cap comment now gives the measured sizes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
Problem and result
Issued forecasts and the quarter-hour measurements they can be scored against live in state.db for about 30 days, then they are gone. This adds a separate
forecast-archive.dbbeside state.db so a box can keep them locally for evaluation later, without putting that work on the planner.The copier reads the existing 30-day issue and observation tables. It does not subscribe to the planner queue. The first run backfills whatever those tables still hold.
Scope and safety
A background worker opens
forecast-archive.dbtwo minutes after start, then once an hour. Boot does not open it and does not fail if it is damaged. Health, storage inventory, backup and shutdown never wait for that open, so a slowquick_checkon an SD card cannot stall/api/health.Core moves the file aside only for real damage:
quick_checkfinds problems, or SQLite returns a corruption code while opening or copying. Busy, permission, I/O and full-disk errors leave the file in place and the worker tries again next hour. After a corruption code during a copy, the next open moves the file aside without askingquick_checkagain, once the last read on the old handle has ended. At most one quarantined copy is kept. Theforecast_archiveblock in/api/healthis informational. It does not change the top-level box status or the boot storage banner, and itslast_erroris a short reason; raw errors with host paths go only to the Core log.Stored series are
champion,planning,legacy_shadowandenergyplan, as little-endian float32 vectors, with model version and a SHA-256 of the frozen site, weather, occupancy and model references. Each run keeps exactly the weather rows it was issued with, including rows from mixed fetches and rows outside the series span. Baselines are not stored; they can be rebuilt from the archived observations and weather.Inside the newest 30 days every run is kept, and model states are kept on one run per UTC hour: the run on the hour, or the first run copied into an hour that has none. The states between can be rebuilt from the archived observations. After 30 days, one run per complete UTC hour is kept, preferring the run that holds states, then an origin on the hour. A run that survived a pass keeps its place, so a later pass or a late copy never splits an hour. Model states are then kept on one run of each complete local day, including DST days and half-hour zones.
The cap is checked on every hourly copy. If live pages pass 1 GiB, the oldest days outside the newest 30 are removed; the newest 30 days are never deleted for the cap, and the archive reports
over_capuntil it fits. Below 1 GiB free on the data volume, copying pauses, but thinning, expiry and the cap still run, and the archive gives back the shortfall from its oldest days outside the newest 30. There is no new config-file setting.The copy is robust to late rows and clock steps: each copy reads again from an hour before its cursor, never reads past the clock plus the 5-minute skew state.db accepts, and resets a cursor that a wrong clock pushed ahead. A row the archive cannot store is skipped with a warning instead of stopping the copy.
Quarter-hour truth is load and PV from
forecast_observations. That table does not record grid power, so grid is not invented here and nothing is backfilled from history.db. Grid belongs with the 5-minute actuals in a later change.A full backup leaves the live archive and its WAL out and stores a
VACUUM INTOcopy instead. A damaged archive is skipped with a warning so it cannot cost the owner the backup. Quarantined files never enter a backup. The quick recovery snapshot is still only state.db.An older Core that does not know this file will ignore it. Its full backup would still copy the live file, including a WAL, because it does not exclude the name. That is accepted for this change.
The archive holds site id, latitude, longitude and occupancy on the box. That is the same class of data as state.db. Backups include it, as they already include state.db. Nothing here uploads it.
Defaults, changeable later
forecast-archive.dbnext to state.db, not opened during bootforecast_archiveblock only, short reasons; box status unchangedSize
The Energyplan state has a fixed shape: 56 days × 72 load buckets and 216 PV cells. Built to that shape it is about 464 KiB of JSON and 87 KiB gzip, and the worker writes a new one about every quarter. Kept on every run, it filled about 250 MiB of the newest 30 days, and 512 MiB held only four to six months. That is why model states are kept once per hour and the cap is 1 GiB.
A 45-day simulation at that state size, with the real legacy model shapes, four 193-point series with bands, weather fetched every 3 hours and daily maintenance (Apple M5, not a Pi):
So two years fit with about 250 MiB to spare. The real Energyplan state on a box was not measured; a larger state grows the newest 30 days by about 2.1 MiB per day for each extra 87 KiB, and barely changes the thinned days.
Left for later
go/internal/forecastingso they can be recomputed./api/forecast/accuracyand the Twins view.Not archived: the per-point
ModelPV/ModelLoadevidence, and bands of series other thanchampion. They load as empty. No reader of the archive uses them today.A Pi has not run the copy benchmark.
BenchmarkForecastLongArchiveCopyis there for that measurement.Verification
All local, on the branch head:
make verify(vet, tests, compose migration, build): passnode --test 'web/**/*.test.mjs': passgo test -raceoninternal/state,internal/backupandcmd/ftw: passEach fix has a test that fails without it, checked by reverting or mutating the fix: mixed-fetch weather round-trip; weather outside the series span; GC and thinning query plans use their indexes; whole-hour thinning at a mid-hour pass in Stockholm and Kolkata; DST days with 23 and 25 runs; a late run loses to a thinned survivor; maintenance and giving space back below the floor; the cap on an hourly copy; status readers do not wait for the open; a permission error does not quarantine; a damaged handle is dropped and healed without a restart; the rebuild waits for a running read; one quarantined copy; late shadow issues and late quarters are copied; a future cursor recovers; rows the archive cannot store are skipped; an archive heal leaves the boot storage event alone; model states are kept once per hour; thinning keeps the hour's state holder when runs arrive out of order.
Checklist
🤖 Generated with Claude Code