Repository navigation
fix(archive): preserve microseconds and pre-epoch SQLite history - #184
TimeToBuildBob wants to merge 17 commits into
Conversation
…_id, id)) Adds `device_id` to the `buckets` table so buckets from different devices can share a bucket name without colliding. Python mirror of ActivityWatch/aw-server-rust#786. **Before:** `UNIQUE(id)` — two devices' `aw-watcher-window_hostname` buckets can't coexist in the same db. **After:** `UNIQUE(device_id, id)` — each device's buckets are scoped independently. Changes: - `LATEST_VERSION` 1 → 2 (new file: `sqlite.v2.db`) - `CREATE_BUCKETS_TABLE`: add `device_id TEXT NOT NULL DEFAULT 'local'`, replace inline `UNIQUE` on `id` with table-level `UNIQUE(device_id, id)` - `create_bucket()`: add `device_id` param (default `"local"`) - `buckets()`, `buckets_with_last_updated()`, `get_metadata()`: include `device_id` - `migration.py`: add `sqlite_v1_to_v2()` to copy v1 data with `device_id='local'`; `check_for_migration()` now detects and runs it on first open of the new db file Part of ActivityWatch/activitywatch#302 (hostname-free bucket IDs). Git-Session-Id: 02ce
- Add device_id param to AbstractStorage.create_bucket() so mypy accepts migration.py's call with device_id="local" (fixes CI typecheck failure) - Add device_id param to PeeweeStorage and MemoryStorage to satisfy override compatibility with the updated abstract signature - Fix sqlite_v1_to_v2(): clear event.id before insert_many so events get INSERTed rather than UPDATE-attempted on an empty destination database (fixes migrated buckets silently having zero events — Greptile P1) - Fix check_for_migration(): prefer SQLite v1 over Peewee; running both into the same destination caused IntegrityError on UNIQUE(device_id, id) when users had both files (Greptile P1) - Add in-place schema upgrade in SqliteStorage.__init__() via ALTER TABLE ADD COLUMN so databases opened via custom filepath= get device_id without failing on SELECT/INSERT queries (Greptile P1) Git-Session-Id: 5ec50786-5c09-5bb0-b4f1-0ffd16b82ac1
…UE(device_id,id) on legacy dbs; commit migration tail Git-Session-Id: 0cec7590-494f-538b-9531-87ebf5dff828
…ned error Git-Session-Id: 9a545a26-3da2-5263-b86a-df4f86f42f20
…ce_id in memory/peewee backends - Migration: skip buckets already present in the v2 datastore and rename the v1 db to *.migrated-v1.db on success so it is not re-detected on next startup (re-running previously raised IntegrityError and blocked startup). - MemoryStorage: include device_id in bucket metadata (API contract parity with the sqlite backend). - PeeweeStorage: report device_id='local' in json() without a schema change (existing peewee DBs have no such column). - Add tests/test_sqlite_migration.py covering migration, rename, and rerun. Git-Session-Id: b4d76fbc-0b40-5323-a560-63560f9e7534
Git-Session-Id: 44e46ae0-002f-5190-958f-74c5450808fa
Git-Session-Id: d421133f-910e-51a5-b6fd-2d6bb107f2e6
Git-Session-Id: d421133f-910e-51a5-b6fd-2d6bb107f2e6
Git-Session-Id: ee485138-49ef-455b-9c15-d08a96a24bd7
Git-Session-Id: ee485138-49ef-455b-9c15-d08a96a24bd7
Git-Session-Id: 9d1828c6-192e-58ea-946a-4405cde28acf
Git-Session-Id: 931478ca-2d2b-48c5-bb99-0bb5ad608478
Open an independent query-only connection that pins a WAL reader snapshot so long-lived exports do not hold the writer connection. Git-Session-Id: aeaa4ffa-df33-4fe0-8487-f35c72996602
Git-Session-Id: 0dc232f4-6721-4652-9e55-bd51fe8eec69
 [High impact] The PR reaches shared event storage and persisted SQLite archives. Fix migration source discovery before merging; WAL companion files can prevent startup. Findings
SummaryThis PR adds device-scoped SQLite access and preserves full Python microseconds.
TimeToBuildBob acknowledged independent creation/snapshot blockers and deferred metadata-repair/alias acceptance checks in the PR description. Those concerns are not repeated as findings. TimeToBuildBob explicitly identified creation and snapshot blockers in #183, including reads that can end a snapshot transaction. TimeToBuildBob also stated that metadata-repair/alias interactions still need the shared acceptance matrix. Diagram%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Open SQLite v2] --> B[Discover SQLite v1 files]
B --> C[Take first matching filename]
C --> D{Main database?}
D -->|Yes| E[Copy buckets and raw event values]
E --> F[Commit data and completion marker]
D -->|WAL or shared-memory file| G[Table read fails]
G --> H[Startup stops]
Reviews (1) · Last reviewed commit: "fix(archive): preserve microseconds and ..." · Reviewed by Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| sqlite_v1_files = detect_db_files(data_dir, sqlite_name, 1) | ||
| if len(sqlite_v1_files) > 0: | ||
| v1_path = os.path.join(data_dir, sqlite_v1_files[0]) | ||
| sqlite_v1_to_v2(datastore, v1_path) |
There was a problem hiding this comment.
detect_db_files accepts sqlite.v1.db, sqlite.v1.db-wal, and sqlite.v1.db-shm. This code takes the first match from unsorted os.listdir output. If a companion file comes first, sqlite_v1_to_v2 opens it as the source database and fails when reading its tables, preventing server startup.
Select the exact main database filename and add a discovery test with companion files listed first.
There was a problem hiding this comment.
Confirmed on current head 4a04a57: running the existing discovery function with WAL/SHM entries before the main file returns all three and selects sqlite.v1.db-wal. This finding remains open. The implementation branch and worktree are currently leased to another session, so I have handed the repair to that owner rather than editing concurrently: select the exact main database, test companion-first ordering and companion-only absence, then refresh the dependent ActivityWatch/aw-server#201 core pin. No fix has been pushed by this monitoring pass.
There was a problem hiding this comment.
Fixed in 538886d. Reproduced with sqlite-testing.v1.db-wal and -shm listed before the main file: discovery returned companions/backups. Discovery now admits only main .db files and exact three-component versioned names. The regression verifies discovery and an actual v1→v2 copy; isolated suite 429 passed / 5 skipped, mypy 37 files. Existing WAL/SHM files are neither opened as sources nor deleted.
Git-Session-Id: 0dc232f4-6721-4652-9e55-bd51fe8eec69
Git-Session-Id: 0dc232f4-6721-4652-9e55-bd51fe8eec69
Git-Session-Id: c2d218fa-690b-4e77-96e3-959267ad65f3
Summary
Archive-fidelity continuation of ActivityWatch/activitywatch#302. Stacked on draft #183 at
4ac2146, which itself depends on schema #177. Do not merge this before those prerequisites; the incremental archive commit is60917e0. The existing implementation branch is leased to another session, so this follow-up does not mutate it.Eventconstruction, timestamp assignment and JSON instead of silently truncating to milliseconds.Reproduction and verification
The initial 21-case fixture failed on the base:
.123457became.123000, old float arithmetic lost additional digits for distant dates, and a 1960 snapshot event vanished. After repair:astimezone(UTC)preserving the previous local-time interpretation, and regression-tested.make test-fast: 1505 passed, 1 skipped;make typecheck: 286 package / 1803 script files passed. These are workspace checks, not aw-core evidence.Boundaries
Python datetime supports microseconds, not Rust's full nanosecond domain. Existing legacy REAL values retain their raw SQL comparison boundaries; sub-microsecond noise cannot be represented exactly by datetime.
Bucket.getkeeps its existing outward millisecond rounding for backend compatibility. No data is rewritten to pretend otherwise.This remains draft: the current #183 has independent known creation/snapshot blockers, and the shared HTTP/migration/same-name/privacy/reader-refresh acceptance matrix is not complete. Both Rust global duplicate-name guards remain. No watcher or V2 rollout, production database changes, or release claim.
Continuation: prerequisite correctness blockers repaired in
4a04a57The leased prerequisite branch remains unchanged. This follow-up now also contains the fixes for all three concrete #183 P1 findings:
BEGIN IMMEDIATEencloses creation alias checks plus INSERT, preventing competing SQLite connections from splitting the two spellings.commit()is a no-op, including through scoped handles; ordinary reads and default iterators retain the pinned view. The owning caller still closes the query-only connection.All six new fixture cases failed before repair. Final isolated aw-core suite 428 passed / 5 skipped, mypy 37 files, pinned Ruff and hooks green. A second bounded read-only review found no concrete correctness issue; the reviewer could not execute Python under its read-only allowlist, so all executable verification is parent-run. Brain second fast/type pass: 1505 passed / 1 skipped, 286/1803 files. #183 itself still contains the old bugs until the implementation owner integrates the two incremental commits; do not resolve its review threads based solely on this sibling draft.
Migration-discovery repair:
538886d(current head)Reproduced the review's startup blocker with WAL/SHM entries listed before
sqlite-testing.v1.db: discovery selected companions and backups.detect_db_filesnow accepts only main.dbfiles and exact three-component versioned names. New test verifies both discovery and the real migration copy. Final suite 429 passed / 5 skipped, mypy 37 files; third brain fast/type pass 1505/1 skipped, 286/1803 files; normal hooks passed. Replied on the finding with the commit and evidence. No WAL/SHM deletion or source data rewrite.