Skip to content

feat(forecast): keep a long-term local forecast archive - #2024

Open
frahlg wants to merge 9 commits into
masterfrom
cursor/forecast-long-archive-7337
Open

frahlg wants to merge 9 commits into
masterfrom
cursor/forecast-long-archive-7337

Conversation

@frahlg

@frahlg frahlg commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Problem and result

Issued forecasts and the quarter-hour measurements they can be scored against live in state.db for about 30 days, then they are gone. This adds a separate forecast-archive.db beside state.db so a box can keep them locally for evaluation later, without putting that work on the planner.

The copier reads the existing 30-day issue and observation tables. It does not subscribe to the planner queue. The first run backfills whatever those tables still hold.

Scope and safety

A background worker opens forecast-archive.db two minutes after start, then once an hour. Boot does not open it and does not fail if it is damaged. Health, storage inventory, backup and shutdown never wait for that open, so a slow quick_check on an SD card cannot stall /api/health.

Core moves the file aside only for real damage: quick_check finds problems, or SQLite returns a corruption code while opening or copying. Busy, permission, I/O and full-disk errors leave the file in place and the worker tries again next hour. After a corruption code during a copy, the next open moves the file aside without asking quick_check again, once the last read on the old handle has ended. At most one quarantined copy is kept. The forecast_archive block in /api/health is informational. It does not change the top-level box status or the boot storage banner, and its last_error is a short reason; raw errors with host paths go only to the Core log.

Stored series are champion, planning, legacy_shadow and energyplan, as little-endian float32 vectors, with model version and a SHA-256 of the frozen site, weather, occupancy and model references. Each run keeps exactly the weather rows it was issued with, including rows from mixed fetches and rows outside the series span. Baselines are not stored; they can be rebuilt from the archived observations and weather.

Inside the newest 30 days every run is kept, and model states are kept on one run per UTC hour: the run on the hour, or the first run copied into an hour that has none. The states between can be rebuilt from the archived observations. After 30 days, one run per complete UTC hour is kept, preferring the run that holds states, then an origin on the hour. A run that survived a pass keeps its place, so a later pass or a late copy never splits an hour. Model states are then kept on one run of each complete local day, including DST days and half-hour zones.

The cap is checked on every hourly copy. If live pages pass 1 GiB, the oldest days outside the newest 30 are removed; the newest 30 days are never deleted for the cap, and the archive reports over_cap until it fits. Below 1 GiB free on the data volume, copying pauses, but thinning, expiry and the cap still run, and the archive gives back the shortfall from its oldest days outside the newest 30. There is no new config-file setting.

The copy is robust to late rows and clock steps: each copy reads again from an hour before its cursor, never reads past the clock plus the 5-minute skew state.db accepts, and resets a cursor that a wrong clock pushed ahead. A row the archive cannot store is skipped with a warning instead of stopping the copy.

Quarter-hour truth is load and PV from forecast_observations. That table does not record grid power, so grid is not invented here and nothing is backfilled from history.db. Grid belongs with the 5-minute actuals in a later change.

A full backup leaves the live archive and its WAL out and stores a VACUUM INTO copy instead. A damaged archive is skipped with a warning so it cannot cost the owner the backup. Quarantined files never enter a backup. The quick recovery snapshot is still only state.db.

An older Core that does not know this file will ignore it. Its full backup would still copy the live file, including a WAL, because it does not exclude the name. That is accepted for this change.

The archive holds site id, latitude, longitude and occupancy on the box. That is the same class of data as state.db. Backups include it, as they already include state.db. Nothing here uploads it.

Defaults, changeable later

Choice Default in this PR
Where forecast-archive.db next to state.db, not opened during boot
Series champion, planning, legacy_shadow, energyplan
Baselines not stored; recompute later
Detail every run for 30 days, then one run per complete UTC hour
Model states one run per UTC hour for 30 days, then one run per complete local day
Retention 2 × 365 days, within the cap
Cap 1 GiB of live pages, checked hourly; oldest days first; never the newest 30 days
Free space below 1 GiB free, copying pauses and the archive gives space back
Damage moved aside only on a corruption code; one quarantined copy kept
Settings none added
Health forecast_archive block only, short reasons; box status unchanged
Truth quarter-hour load and PV; no grid; no history.db backfill
Backup consistent copy in a full backup; omitted from the quick snapshot

Size

The Energyplan state has a fixed shape: 56 days × 72 load buckets and 216 PV cells. Built to that shape it is about 464 KiB of JSON and 87 KiB gzip, and the worker writes a new one about every quarter. Kept on every run, it filled about 250 MiB of the newest 30 days, and 512 MiB held only four to six months. That is why model states are kept once per hour and the cap is 1 GiB.

A 45-day simulation at that state size, with the real legacy model shapes, four 193-point series with bands, weather fetched every 3 hours and daily maintenance (Apple M5, not a Pi):

Measure Result
Newest 30 days, full detail 160 MiB (series 74, model states 72, runs 11)
One thinned day 0.87 MiB
30 days + 700 thinned days about 770 MiB, 75% of the 1 GiB cap
Hourly copy of 4 runs (archive write) median 3.9 ms, p95 9.1 ms
Daily maintenance from day 31 about 1.3 s, most of it batch pauses and GC

So two years fit with about 250 MiB to spare. The real Energyplan state on a box was not measured; a larger state grows the newest 30 days by about 2.1 MiB per day for each extra 87 KiB, and barely changes the thinned days.

Left for later

  • PR 2: qualified 5-minute actuals, including grid. No backfill from history.db unless that decision changes.
  • PR 3: RMSE, skill and an evaluate path over this file. Baselines move into go/internal/forecasting so they can be recomputed.
  • PR 4: /api/forecast/accuracy and the Twins view.
  • PR 5: walk-forward replay.
  • PR 6–8: research consent, upload and training. Not in this diff.

Not archived: the per-point ModelPV/ModelLoad evidence, and bands of series other than champion. They load as empty. No reader of the archive uses them today.

A Pi has not run the copy benchmark. BenchmarkForecastLongArchiveCopy is there for that measurement.

Verification

All local, on the branch head:

  • make verify (vet, tests, compose migration, build): pass
  • node --test 'web/**/*.test.mjs': pass
  • go test -race on internal/state, internal/backup and cmd/ftw: pass

Each fix has a test that fails without it, checked by reverting or mutating the fix: mixed-fetch weather round-trip; weather outside the series span; GC and thinning query plans use their indexes; whole-hour thinning at a mid-hour pass in Stockholm and Kolkata; DST days with 23 and 25 runs; a late run loses to a thinned survivor; maintenance and giving space back below the floor; the cap on an hourly copy; status readers do not wait for the open; a permission error does not quarantine; a damaged handle is dropped and healed without a restart; the rebuild waits for a running read; one quarantined copy; late shadow issues and late quarters are copied; a future cursor recovers; rows the archive cannot store are skipped; an archive heal leaves the boot storage event alone; model states are kept once per hour; thinning keeps the hour's state holder when runs arrive out of order.

Checklist

  • The change follows VISION.md and one selected scope.
  • I checked overlapping PRs and coordinated shared files/contracts.
  • Relevant checks cover the changed behaviour and failure paths.
  • A human reviewed changed UI in a browser, or no UI changed.
  • A Changeset is included, or the change is exempt.
  • Every commit has a DCO sign-off.

🤖 Generated with Claude Code

Copy the 30-day forecast tables into forecast-archive.db after startup.
The file stays off the boot and planning path, keeps full detail for 30
days, then one run per hour for two years, and stops before a 512 MiB
cap or a full data volume. Health reports it without changing box status.
Full backups store a consistent copy.

Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T19:20:24.597758Z daed653 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: daed653b55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1180 to +1181
query := `SELECT source, available_at_ms, start_ms, end_ms, ghi, cloud, temp, direct_pv, est_pv FROM fa_weather
WHERE available_at_ms IN (` + placeholders(len(ids)) + `) AND start_ms < ? AND end_ms > ? ORDER BY start_ms, source`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve exact weather-row membership for each archived run

When a forecast uses a mixture of fetch timestamps—for example, when a partial provider refresh leaves an older cached slot—this query retrieves every overlapping row from both fetches, not just the rows captured by that run. Rows from the older fetch that were replaced in the issue are consequently added back alongside their replacements. I reproduced this with two valid archived issues: ArchiveIssues succeeds, but LoadIssue then fails validation with weather unavailable at forecast origin because the reconstructed weather intervals overlap. Store each run's exact weather-row references rather than only its availability timestamps.

Useful? React with 👍 / 👎.

Comment on lines +71 to +72
rows, err := a.db.QueryContext(ctx, `SELECT run_id, origin_ms, timezone FROM fa_runs
WHERE thinned=0 AND origin_ms<? ORDER BY origin_ms, run_id`, cutoff)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include previously retained winners when thinning cutoff batches

When daily maintenance runs partway through an hour, the 30-day cutoff splits that hour across maintenance passes. The first pass marks its retained run as thinned; the next pass excludes that winner here and independently retains another run from the same UTC hour. I reproduced this with runs at 12:00, 12:15, 12:30 and 12:45 and successive daily cutoffs at 12:02: two runs remain permanently instead of one. This also retains an additional model-state snapshot for the same local day. Compare newly eligible runs against previously retained winners, or process only complete expired hours.

Useful? React with 👍 / 👎.

frahlg and others added 8 commits November 16, 2026 09:53
The archive linked a run only to the fetch times it used and, on load,
returned every row of those fetches that overlapped the series span. After
a partial weather refresh one issue holds rows from two fetches, so another
run's row for the same slot came back too and the issue failed validation.
Rows outside the span were dropped.

fa_run_weather now holds one row per run and fetch, keyed by run, source
and fetch time, with the run's slot starts packed as uvarint gaps. LoadIssue
reads exactly those rows, all of them, in start order. The fa_runs
weather_avail column is gone. The fa-v1 schema changes in place; no box
has the file yet.

Garbage collection probed fa_run_weather by fetch time with no index, so
every candidate row scanned all links. A new index on (source,
available_at_ms) serves the weather probe, and the model-state probe uses
fa_run_states_ref. A test checks both query plans.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
…floor

Each thinning pass split the hour and the local day at the cutoff, so the
rest of a split hour kept a second run and a split day kept a second set
of model states. In half-hour zones every hour at a day boundary kept two
runs. Below the disk floor the deletes refused to run, so nothing was
thinned, expired or pruned, and the cap waited for the daily pass.

Thinning now works on complete UTC hours. It ranks every run of the hour,
keeps a thinned survivor if there is one, and so drops a run copied late
into a thinned hour. Model states stay on the first kept run of each
complete local day that has them. A partial index finds unthinned runs.

Deletes no longer check the floor. Below it, copying stays paused but
maintenance runs, and the cap target drops by the missing free space, so
the oldest days outside the newest 30 give that space back. Every Sync
enforces the cap, and incremental vacuum ends with a checkpoint so the
file shrinks at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
…l damage

The worker held the store's archive lock while it opened the file. A
quick_check of a large file on an SD card made /api/health, the storage
inventory, backups and shutdown wait behind it. The open now runs
without that lock. A second lock keeps two opens apart, and a store that
closes in the meantime gets no handle.

Any quick_check error moved a good file aside, so a passing I/O, memory
or full-disk error cost the whole archive. Now only a check that finds
problems, or a corruption code, does that. Other errors report
unavailable and the worker tries again next hour. Each heal removes
older quarantined copies, so at most one stays. Backups still leave
them out.

Damage found while copying waited for a restart. Sync now closes the
handle when the archive returns a corruption code, so the next hourly
run checks and heals the file. A corruption code read from state.db
does not count.

The worker gave the open 30 seconds. A full quick_check can take longer
on a slow card, and then every hourly try fails the same way. The open
is off the boot path and nothing waits on it, so it now runs until it
ends or shutdown cancels the worker's context.

Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The copier read strictly after its cursor. A shadow issue saved just
after its primary, with the same issued time and an id that sorts
first, was skipped for good. Observations were read in start order,
so a quarter released after a later one was skipped too. A cursor set
while the clock ran ahead stopped all copying until that time came.

Observations now follow available_at_ms, the order the tracker writes
them in. Each copy reads again from an hour before its cursor and
never past the clock plus the five minutes state.db accepts, so a row
stamped by a wrong clock cannot pull the cursor ahead. A cursor already
past that bound resets, and the copy reads the 30-day tables again.

A rescan stays cheap: the copier lists issue keys without payloads and
loads only the runs the archive does not hold. Rows read again are
ignored on insert. The cursor and last_copy_ms move only when the copy
gets past the old cursor or stores a run. The fa-v1 cursor changes in
place; an old cursor reads as zero and starts one full rescan.

Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three guards in the archive had no test that would fail without them.
The daily thinning tests ran 2 minutes past the hour, when the open hour
held only its run on the hour, so a pass that thinned a partial hour
still passed. They now run at 18:40 UTC: the open hour holds runs a bad
pass would delete, and a Kolkata day has just ended inside an open hour.
A plan test now checks that the thinning queries read fa_runs through
fa_runs_unthinned and fa_runs_origin. Another test deletes a linked
weather row and expects LoadIssue to fail.

The cleanup removes code nothing used. The delete helpers lose their
one-value table and column arguments, forecastLongDay loses its unread
key, NewestForecastIssueMS goes, and the observation reader returns a
slice with no nil check or limit clamp, as the issue reader does.
incrementalVacuum checkpoints only at the end: the vacuum sees pages
freed in the WAL, and only the last checkpoint shrinks the file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
- Skip an issue or observation the archive cannot store, with a warning,
  instead of stopping the copy at that row until state.db prunes it.
- After a corruption code, move the file aside on the next open without
  asking quick_check again, and wait until the last read on the dropped
  handle has ended, so its close cannot delete the new file's WAL.
- Health shows a short reason instead of raw errors with host paths, and
  an archive heal no longer replaces the boot storage event.
- Log model states that state.db pruned before the copy.
- Name forecast-archive.db in the operations and backup docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
… 1 GiB

The Energyplan state has a fixed shape (56 days of 72 load buckets and
216 PV cells), about 450 KiB of JSON and 75-100 KiB gzip, and the worker
writes a new one about every quarter. Kept on every run, it filled about
250 MiB of the newest 30 days, so 512 MiB held only four to six months.

Inside the newest 30 days a run now keeps its model states only when it
is on the hour, or when it is the first run copied into an hour that has
none. Thinning already keeps the run on the hour, and the states between
can be rebuilt from the archived observations. With the cap at 1 GiB, two
years fit by the same estimate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
In an hour with no run on the hour, the first run copied holds the
model states. A run issued earlier but copied later used to win the
hour, so the holder was deleted and the day could lose its states.
Hour thinning now ranks a run with states first. The plan test covers
the new hour query and checks fa_run_states is never scanned. The cap
comment now gives the measured sizes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants