[iandees]

Why

The origin VM was pegged on CPU (netdata alerts). Cloudflare analytics show most uncached /1.0/data/show/* traffic that reaches the origin is 400s averaging 2-3s of origin time each, plus a few dozen requests that run ~125s and end as Cloudflare 524s.

The 400s come from requests with 10-15 sumlevel|parent_geoid groups (e.g. 040|01000US,950|01000US,150|01000US,...). Each group was expanded against Postgres, and for latest the whole thing was repeated for every release, before the request was finally rejected for exceeding MAX_GEOIDS_TO_SHOW.

Changes

• Fail fast in expand_geoids: takes the endpoint's max_geoids, dedupes repeated entries, and aborts with a 400 as soon as the running total goes over the limit. Used by data/show, data/download and geo/show.
• KNOWN_LARGE_EXPANSIONS lookup: expansions we already know are huge (e.g. block groups or tracts for the nation, block groups for large states) are rejected without querying. Counts are compared against the configured limit, apply to all releases, and are documented as an optional refresh in DATA_UPDATES.md.
• 5s Postgres statement timeout for interactive endpoints via SET LOCAL in a before_request hook (so it can't leak through the connection pool). /1.0/data/download/, /1.0/aggregate/ and /1.0/user_geo/ are exempt, and INTERACTIVE_STATEMENT_TIMEOUT_MS = 0 disables it. A cancelled query now returns a JSON 503 instead of a 500.

Things to watch after deploy

• The 5s timeout will turn some currently-slow-but-successful requests (large data/show, geo/show, geo/search, table/search) into 503s. Check the 503 rate and raise INTERACTIVE_STATEMENT_TIMEOUT_MS if it's too aggressive.
• The error text for over-limit requests changed to "You requested too many geoids. The maximum is N..." when caught early.

Testing

Unit tests for the early abort, dedupe, known-large lookup, timeout hook and 503 handler (35 pass with the existing tests). Not yet exercised against a real database.