Repository navigation
Help complete the community database: opt-in unknown-ASIN report (LibraForge C3) - #313
Merged
Merged
Conversation
…okup and the intake POST
An opt-in switch (default off, AUDIOSILO_METADATA_REPORT_UNKNOWN), the table
the daily report keeps its lookups and sends in (no book or path), the
candidate rule (own ASINs except community ones, single-book tag hints), and
meta.Service.KnownASIN (only a 404 is unknown) and ReportUnknownASINs: https or
loopback only, no redirects, a client SetMirror never touches, and a body of
exactly {"asins": [...]} with no identifying headers.
Once a day while metadata and report_unknown are on: Ping, up to 500 lookups of the candidates due (30-day recheck, shuffled), then the unknown ones in chunks of 200 (30-day resend, shuffled, once a day by MAX(sent_at)). A 404, 405, 3xx or odd 2xx is not_offered with a week's quiet; 429/5xx mark nothing. Mirror mode waits for the local copy and still POSTs to the remote service. Never built on a demo server. GET /admin/settings carries metadata.report_status; switching it on wakes the job.
Settings > Metadata gains the report_unknown switch (disabled while metadata is off, hidden from an older server) and the last run in words: not sent yet, sent, nothing new, not taken by the service, or failed. Every locale.
One JobRow for the upcoming and report switches; ReportUnknownASINs returns only its error; run returns its failure instead of a stored string (last_error dropped); On is required; check computes its own stamp; the row map no longer repeats its key.
The schedule no longer spins while the report is off past its run time; KnownASIN counts a persistent-store hit (CachedWorkID) as known; catalog.ParseStamp beside FormatStamp for coverage and upcoming; the mirror-mode wording says a lookup the copy fails still falls back to the remote service.
KnownASIN marks its lookup local-only: fallbackTransport answers it from the local copy or fails it (errLocalOnly) when the copy fails it or is not Ready, never asking the remote service. Every other mirror-mode request still falls back. Tests: the copy answering 500, timing out and dropped mid-run make no remote GET and stop the checking; a non-report lookup still falls back.
A Ping the local copy fails falls back to the remote service; in mirror mode the copy being ready is the check, so the run skips it.
This was referenced Nov 6, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LibraForge Phase C, sub-phase C3: an opt-in daily report of the Audible ASINs the community database does not know, so the meta-sync reported-ASIN lane (C2, live: meta#2895 merged its first 5 works) can add them.
What
metadata.report_unknown(default off, envAUDIOSILO_METADATA_REPORT_UNKNOWN), Settings > Metadata switch "Help complete the community database" with a status line (report_status: never / ok / not_offered / failed), all 6 locales. Disabled while metadata is off; never on a demo server.internal/coverage: daily job. Candidates arebooks.asin(not community-override ASINs) plus tag ASINs only on books with no ASIN/ISBN, no ASIN edit, and a tag on exactly one book. Each is looked up first; only a 404 is unknown, any other error stops checking. Bounds: 500 checks a run, 30-day recheck/resend, shuffled, chunks of 200, at most 5 POSTs, once a day across restarts.{"asins":[...]}with Content-Type andUser-Agent: AudioSiloonly. It goes over https only (plain http only to loopback), never follows a redirect, and treats 404/405/3xx/2xx-without-acceptedas not offered (7 quiet days). 429/5xx marks nothing.unknown_asin_reports(asin, checked_at, unknown, sent_at), no path stored; run status inserver_state.Tests: wire (exact headers and body keys), every answer code, no-redirect, http off-machine, mirror local-only (500 / timeout / copy dropped mid-run, zero remote GETs), off and metadata-off make zero calls, bounds, once a day across restart, prune, candidate SQL table, migration test, console tests.
Review: /simplify + /code-review xhigh --fix (fixed a scheduler spin while the switch is off), then one fix round at high on the mirror local-only change.
Verified live (local metaserve at meta origin/main with intake on, a capture proxy, Playwright): switch default off and disabled with metadata off; status never -> ok; the captured POST held only the 4 unknown ASINs, upper-cased, no shared-tag or known ASIN, minimal headers; queue GET with the token showed them; mirror mode sent only the POST (no lookups, no healthz); intake off gave not_offered and no calls on a re-run; a 503 POST and a metaserve killed mid-run marked nothing; a LAN http base_url made no request. Not live: the redirect path and >200-ASIN chunking (unit tests only), real https.
Pairs with KodeStar/audiosilo-docs (docs), KodeStar/audiosilo-site (privacy copy, held for the release), KodeStar/audiosilo-meta (privacy page).