[jamesray]

The gap

import_GOES_data_to_timestream runs hourly and slices exactly one hour back from Time.now():

last_hour = Time.now() - TimeDelta(1 * u.hour)
tsa_last = tsa.loc[last_hour : Time.now()]

NOAA publishes each minute a few minutes late. The rule fires at :54:56, so the newest point in the feed at that moment is around :49. The minutes that arrive after the run starts are never picked up, because the next run's window begins at :54:56 — already past them.

Timestream shows the hole. Sample counts per minute-of-hour over 24 hours:

Minute of hour	Samples
:00–:49	23–24 each
:50	13
:51	6
:52–:54	0
:55–:59	23 each

That is roughly two hours of GOES X-ray flux per day that never lands, every day.

The change

Look back two hours instead of one, so every minute falls inside two consecutive runs.

This costs nothing extra:

• the feed is xrays-6-hour.json, so six hours already come back on every call — no additional request
• Timestream deduplicates records with identical dimensions, measure name and timestamp
• record_timeseries catches RejectedRecordsException and logs it rather than raising (swxsoc/db/timeseries.py:377), so a revised value that loses a version check cannot fail the job

It also makes the job self-healing: a run that fails now gets backfilled by the next hour instead of leaving a permanent hole.

Test

Added test_import_GOES_data_covers_the_minutes_between_runs, which stubs the feed with one point 70 minutes old and one 5 minutes old and asserts both are recorded. It fails on the old one-hour window (records 1 of 2) and passes on the new one. No network access.

Verified locally: the new test fails before the change and passes after. Full suite is 8 passed, with test_import_stix_to_timestream and test_get_padre_orbit_data failing only because padre_craft and stixdcpy were not installed in the local environment — neither touches this code path and CI installs the full requirements.

Not included

Separately, goes_flux currently plots as a gap on dashboards roughly 08:30–09:30 UTC daily. That is not a pipeline fault: the stored values are exactly 0.0 during satellite eclipse, and a log-scale axis cannot render zero. The window has been lengthening toward the September equinox (51 zero-minutes on 09-11, 59 on 09-16) and will taper off through October. Worth handling as a null in the panel query rather than here.

[Copilot]

🟢 Approval recommended

The changes address delayed measurements with regression coverage and no unresolved blocking issues.

Pull request overview

Extends the GOES import lookback to two hours to capture delayed NOAA measurements.

Changes:

• Adds an overlapping two-hour import window.
• Adds regression coverage for delayed points.

File summaries

File	Summary
lambda_function/tests/test_executor.py	Tests recording stale and recent measurements.
lambda_function/src/executor/executor.py	Implements the two-hour GOES import window.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.