Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/forecast-long-archive.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"ftw": patch
---

Keep issued forecasts and quarter-hour load and PV measurements in a local archive for up to two years, separate from planning. The archive keeps every run for 30 days, then one run per hour, and drops its oldest days to stay under 1 GiB.

The archive is informational. A problem in it does not mark the box degraded, and a full backup stores a consistent copy rather than the live file.
4 changes: 4 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,10 @@ SQLite history.db owns bucketed history, dashboard summaries and the energy
ledger. A separate state.db owns goals, device identity, learned state and
planner diagnostics. Prices and forecasts live in cache.db; full backups
keep its past prices, which savings history needs.
Issued forecasts and quarter-hour measurements also accumulate in
forecast-archive.db, opened after startup and kept out of the control path.
A full backup stores a consistent copy of that file. A problem there does
not mark the box degraded.
Database access stays in
[`go/internal/state`](../go/internal/state).

Expand Down
4 changes: 3 additions & 1 deletion docs/backup-and-restore.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ known-good copy.

The archive leaves out other backups and old rollback points. It carries a
consistent copy of `cache.db`: savings history is costed from its past
prices, and a price provider sends only today and tomorrow. Managed-driver
prices, and a price provider sends only today and tomorrow. It also carries
a consistent copy of `forecast-archive.db`; if that copy fails its check,
the backup goes on without it and Core logs a warning. Managed-driver
links that point inside the persistent directory become relative links in the
archive, so restore can move the data to another directory or machine.
Backup never follows these
Expand Down
8 changes: 6 additions & 2 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,9 @@ holds:
90 days, 1 h for 5 years) and the energy ledger;
- `cache.db` — prices and forecasts; past prices cost the savings history
and come back only from a full backup;
- `forecast-archive.db` — issued forecasts and quarter-hour measurements
kept for later analysis, at most 1 GiB; Core moves a damaged file aside
and starts an empty one;
- key files such as `nova.key` — the box's identity; keep them with the data;
- `drivers/` — custom and managed drivers;
- `backups/` — full backups, unless `state.backup_dir` points elsewhere.
Expand All @@ -95,8 +98,9 @@ remain on the same disk and cannot recover a failed SD card.

Core exposes a read-only storage check at
`GET /api/storage/inventory`. It reports allocated, live and free SQLite pages
for `state.db` and `cache.db`, their WAL/SHM sizes, and free space on the volume
that contains the data directory. It does not expose paths, scan a storage
for `state.db` and `cache.db`, and for `forecast-archive.db` once Core has
opened it, their WAL/SHM sizes, and free space on the volume that contains
the data directory. It does not expose paths, scan a storage
engine's internal files, change retention, checkpoint WAL, run `VACUUM`, or
delete data. `GET /api/backups` lists full backups and their sizes.

Expand Down
109 changes: 109 additions & 0 deletions go/cmd/ftw/forecast_long_archive.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
package main

import (
"context"
"log/slog"
"sync"
"time"

"github.com/srcfl/ftw/go/internal/state"
)

// forecastLongArchiver copies the 30-day forecast tables into
// forecast-archive.db. It does not share the tracker's queue or its lock.
type forecastLongArchiver struct {
store *state.Store
cancel context.CancelFunc
wg sync.WaitGroup
startDelay time.Duration
interval time.Duration
now func() time.Time
}

func startForecastLongArchive(ctx context.Context, st *state.Store) *forecastLongArchiver {
a := &forecastLongArchiver{store: st, startDelay: 2 * time.Minute, interval: time.Hour}
a.Start(ctx)
return a
}

func (a *forecastLongArchiver) Start(ctx context.Context) {
if a == nil || a.store == nil || a.cancel != nil {
return
}
workerCtx, cancel := context.WithCancel(ctx)
a.cancel = cancel
a.wg.Add(1)
go func() {
defer a.wg.Done()
a.run(workerCtx)
}()
}

func (a *forecastLongArchiver) Stop() {
if a == nil || a.cancel == nil {
return
}
a.cancel()
a.wg.Wait()
}

func (a *forecastLongArchiver) run(ctx context.Context) {
a.store.ReportForecastLongArchive(state.ForecastLongArchiveStatus{
State: "starting", CapBytes: state.ForecastLongMaxBytes,
})
delay := a.startDelay
if delay < 0 {
delay = 0
}
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return
case <-timer.C:
}
a.runOnce(ctx)
interval := a.interval
if interval <= 0 {
interval = time.Hour
}
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
a.runOnce(ctx)
}
}
}

func (a *forecastLongArchiver) runOnce(ctx context.Context) {
if a == nil || a.store == nil || ctx.Err() != nil {
return
}
// No deadline beyond the worker's own context. quick_check of a full
// 1 GiB archive can take minutes on a slow SD card; a fixed timeout
// would throw that work away and fail the same way every hour. The open
// is off the boot path, health does not wait on it, and shutdown cancels
// it through ctx.
arch, err := a.store.OpenForecastLongArchive(ctx)
if err != nil {
if ctx.Err() != nil {
return
}
slog.Warn("forecast long archive unavailable", "err", err)
a.store.ReportForecastLongArchive(state.ForecastLongArchiveStatus{
State: "unavailable", CapBytes: state.ForecastLongMaxBytes, LastError: state.ForecastLongOpenFailed,
})
return
}
now := time.Now().UTC()
if a.now != nil {
now = a.now().UTC()
}
if err := arch.Sync(ctx, a.store, now); err != nil && ctx.Err() == nil {
slog.Warn("forecast long archive copy", "err", err)
}
}
122 changes: 122 additions & 0 deletions go/cmd/ftw/forecast_long_archive_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
package main

import (
"context"
"path/filepath"
"runtime"
"testing"
"time"

"github.com/srcfl/ftw/go/internal/forecasting"
"github.com/srcfl/ftw/go/internal/state"
)

func forecastingIssueForArchive(origin time.Time) forecasting.Issue {
start := origin.UnixMilli()
band := forecasting.Band{Method: forecasting.BandMethodColdStart}
point := forecasting.Point{
StartMS: start, EndMS: start + int64(time.Hour/time.Millisecond),
PVW: 1, LoadW: 2, PVKnown: true, LoadKnown: true,
PVQuality: "ready", LoadQuality: "ready",
PVBand: band, LoadBand: band, NetBand: band,
}
return forecasting.Issue{
Schema: forecasting.Schema, ID: "issued", DecisionID: "decision",
OriginMS: start, IssuedAtMS: start, LatestInputMS: start, ConfigVersion: "cfg",
Series: []forecasting.Series{{Name: "champion", ModelVersion: "v1", Points: []forecasting.Point{point}}},
}
}

func TestForecastLongArchiverStops(t *testing.T) {
st, err := state.Open(filepath.Join(t.TempDir(), "state.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
before := runtime.NumGoroutine()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
archiver := &forecastLongArchiver{store: st, startDelay: 0, interval: time.Hour}
archiver.Start(ctx)
archiver.Stop()
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) && runtime.NumGoroutine() > before {
time.Sleep(20 * time.Millisecond)
}
if got := runtime.NumGoroutine(); got > before {
t.Fatalf("goroutines leaked: before %d after %d", before, got)
}
}

func TestForecastLongArchiverPausesWhenDiskIsFull(t *testing.T) {
st, err := state.Open(filepath.Join(t.TempDir(), "state.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
if err := st.InitForecastArchive(context.Background()); err != nil {
t.Fatal(err)
}
origin := time.Now().UTC().Add(-time.Hour).Truncate(time.Hour)
issue := forecastingIssueForArchive(origin)
if err := st.SaveForecastIssue(context.Background(), issue); err != nil {
t.Fatal(err)
}
restore := state.ForecastLongDiskAvailForTest(func(string) (int64, error) { return 0, nil })
defer restore()
archiver := &forecastLongArchiver{store: st, startDelay: time.Hour, interval: time.Hour}
archiver.runOnce(context.Background())
status, ok := st.ForecastLongArchiveStatus()
if !ok || status.State != "paused_disk_full" || status.Runs != 0 {
t.Fatalf("status = %+v ok=%v", status, ok)
}
}

func TestForecastRecordDoesNotWaitOnLongArchive(t *testing.T) {
st, err := state.Open(filepath.Join(t.TempDir(), "state.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
origin := time.Now().UTC().Add(-time.Hour).Truncate(time.Hour)
if err := st.InitForecastArchive(context.Background()); err != nil {
t.Fatal(err)
}
if err := st.SaveForecastIssue(context.Background(), forecastingIssueForArchive(origin)); err != nil {
t.Fatal(err)
}
entered := make(chan struct{})
release := make(chan struct{})
restore := state.ForecastLongDiskAvailForTest(func(string) (int64, error) {
select {
case <-entered:
default:
close(entered)
}
<-release
return 1 << 40, nil
})
defer restore()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
archiver := &forecastLongArchiver{store: st, startDelay: 0, interval: time.Hour}
archiver.Start(ctx)
select {
case <-entered:
case <-time.After(5 * time.Second):
t.Fatal("copier did not reach a write")
}
f := trackerFixture(origin)
f.store = st
in := f.Snapshot(origin, trackerWeather(origin.Add(time.Minute), origin))
slots := trackerSlots(origin.Add(time.Minute), 1)
started := time.Now()
for i := 0; i < 30; i++ {
in.Record(slots, slots, nil, "decision", origin.UnixMilli())
}
if time.Since(started) > time.Second {
t.Fatal("planner record waited on the long archive")
}
close(release)
archiver.Stop()
}
4 changes: 4 additions & 0 deletions go/cmd/ftw/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -1667,6 +1667,10 @@ func main() {
} else {
defer forecastTrackerSvc.Stop()
}
// The long-term archive copies the 30-day tables on its own. It starts
// even when the tracker did not, and it never shares the tracker's queue.
forecastLongArchiveSvc := startForecastLongArchive(ctx, st)
defer forecastLongArchiveSvc.Stop()

// A small live-identity poll also protects legacy learners when MPC is off.
// Config/schema/script work stays in Configure; this only reads host identity.
Expand Down
8 changes: 8 additions & 0 deletions go/internal/api/api.go
Original file line number Diff line number Diff line change
Expand Up @@ -791,6 +791,9 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
if s.deps.State != nil {
storage := map[string]any{"state": "ok", "cache": "ok", "history": "ok"}
for _, ev := range s.deps.State.HealEvents() {
if ev.Tier == "forecast_archive" {
continue // shown in forecast_archive; it must not replace a boot event
}
storage[ev.Tier] = ev.Action // "rebuilt" | "restored"
storage["last_event_ms"] = ev.AtMs
storage["detail"] = ev.Detail
Expand All @@ -803,6 +806,11 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
if writer.LastError != "" || writer.MaintenanceError != "" || s.deps.State.HistoryMaintenanceStatus().LastError != "" || (writer.LastRejectMS > 0 && time.Now().UnixMilli()-writer.LastRejectMS < time.Minute.Milliseconds()) {
resp["status"] = "degraded"
}
// The long-term forecast archive is informational. A full disk or a
// rebuilt file must not mark the box degraded.
if archive, ok := s.deps.State.ForecastLongArchiveStatus(); ok {
resp["forecast_archive"] = archive
}
}
writeJSON(w, 200, resp)
}
Expand Down
46 changes: 46 additions & 0 deletions go/internal/api/api_health_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,32 @@ func TestHealthStorageReportsCacheRebuild(t *testing.T) {
}
}

func TestHealthForecastArchiveIsInformational(t *testing.T) {
st, err := state.Open(filepath.Join(t.TempDir(), "state.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
body := getHealth(t, st)
if _, ok := body["forecast_archive"]; ok {
t.Fatal("health published an archive block before the copier started")
}
if body["status"] != "ok" {
t.Fatalf("status = %v", body["status"])
}
st.ReportForecastLongArchive(state.ForecastLongArchiveStatus{
State: "unavailable", LastError: "forecast archive rebuilt", CapBytes: state.ForecastLongMaxBytes,
})
body = getHealth(t, st)
if body["status"] != "ok" {
t.Fatalf("archive failure changed box status to %v", body["status"])
}
block, ok := body["forecast_archive"].(map[string]any)
if !ok || block["state"] != "unavailable" {
t.Fatalf("forecast_archive = %#v", body["forecast_archive"])
}
}

func checkpoint(t *testing.T, path string) {
t.Helper()
db, err := sql.Open("sqlite", path)
Expand Down Expand Up @@ -110,3 +136,23 @@ func corruptFileAt(t *testing.T, path string, offset int64) {
t.Fatal(err)
}
}

func TestHealthArchiveHealKeepsBootStorageEvent(t *testing.T) {
st, err := state.Open(filepath.Join(t.TempDir(), "state.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
st.RecordHealEvent(state.HealEvent{Tier: "state", Action: "restored", AtMs: 100, Detail: "state.db restored"})
st.RecordHealEvent(state.HealEvent{Tier: "forecast_archive", Action: "rebuilt", AtMs: 200, Detail: "archive rebuilt"})
storage, ok := getHealth(t, st)["storage"].(map[string]any)
if !ok {
t.Fatal("no storage object in /api/health")
}
if storage["state"] != "restored" || storage["detail"] != "state.db restored" || storage["last_event_ms"] != float64(100) {
t.Fatalf("archive heal replaced the boot event: %v", storage)
}
if _, ok := storage["forecast_archive"]; ok {
t.Fatalf("storage carries the archive tier: %v", storage)
}
}
Loading
Loading