A waiting request says why it is not placed, on a worker #636

Part of #623.

Scope

On a single host (a worker), a request waiting in the queue says why it is not placed yet: in simlock list --requests, GET /v1/lease-requests, status.get waiting, and a "Why" row in the console's Waiting inspector. A worker's NO_CAPACITY answer names the capacity check that refused it, so #637 can show it on a gateway.

A request whose new device fails to provision twice no longer waits: it is rejected with a new error code PROVISION_FAILED carrying the driver's error (decided by the maintainer, 2026-10-18).

In short

Each time the daemon tries to place a waiting request, it keeps the reason the attempt stopped, in memory. A queued request shows the reason from its latest attempt. A starting request shows none: its stage already says the daemon is working on it. The reasons and their texts are feature #623 Shared values, rows "Waiting reason (worker)", "Reason text", "NO_CAPACITY detail" and "PROVISION_FAILED" (decided by the maintainer, 2026-10-18).

flowchart LR
  A[Request arrives or queue wakes] --> P{Plan}
  P -- grant / boot / create / evict --> S[starting]
  P -- blocked --> R["Keep the reason on the waiter (new)"]
  R --> Q[queued]
  Q --> L["list.get requests: reason on the entry (new)"]
  P -- blocked, noWait --> N["NO_CAPACITY with details.capacityReason (new)"]

Words used:

• Reason: the first check that stopped the request at its latest placement attempt, for example platform-running-limit, printed "Full: running limit for iOS".
• Capacity reason: one of the four capacity refusals (device-limit, ram-budget, global-running-limit, platform-running-limit).

Before and after:

$ simlock list --requests
Request req_7: agent-b, ios iPhone 16, queued at 1, waiting 12s
Request req_9: agent-c, ios iPhone 16, queued at 2, waiting 3s
Request req_11: agent-d, android Pixel 8, starting, waiting 1s

$ simlock list --requests          # after
Request req_7: agent-b, ios iPhone 16, queued at 1, waiting 12s — full: running limit for iOS
Request req_9: agent-c, ios iPhone 16, queued at 2, waiting 3s — behind an earlier request
Request req_11: agent-d, android Pixel 8, starting, waiting 1s

If this goes wrong, a user would see a reason that is stale or wrong, or a reason on a request that is starting.

Technical spec

Facts

What changes

1. The plan says why it is blocked. blocked (F7) takes a reason and returns it on wait and no-capacity. Each path gives:

   Path	Reason
   Not the head of the virtual line (F1)	behind-earlier-request
   Warm pool booting a fit (F3)	waiting-for-device
   Boot refused, no victim (F4)	the refusal's capacity reason
   New device refused, no victim (F5)	the refusal's capacity reason
   Victim found, its claim taken (F5, F6)	the capacity reason that asked for the eviction
   Physical, only fits held for an older request (F8)	behind-earlier-request
   Boot claim already taken (F4)	waiting-for-device
   Physical, no fit free (F8)	no-free-physical

   The evict-running and evict-managed plans also carry the capacity reason that asked for them (F6). The capacity reasons map one to one (F9). The mapping is a Record keyed by core's CapacityRefusalReason, so the compiler fails when the two lists differ.

2. The waiter keeps it. Every enqueue sets the waiter's reason from the plan that led to it. Each new plan replaces it. It is never written to state.json (F16).

   • Failed eviction (F12): the capacity reason that asked for the eviction.
   • Failed physical grant (F13): no-free-physical.
   • A request whose match is being cleaned (F14) keeps the capacity reason its plan gave; the reclaiming push stays as today.
   • The head leaving without a wake (F34): cancelling a queued waiter now wakes the queue, so the new head plans again and replaces its behind-earlier-request (decided by the maintainer, 2026-10-18).

3. The listing shows it. waitingRequests() adds reason to each queued entry that has one, never to a starting one (F17). The gateway's own entries get none (F18; A waiting request says why each worker cannot take it, on a gateway #637 adds workers). A gateway's list keeps the reason on a worker's own entries (F35); the CLI prints them with the same suffix.

4. NO_CAPACITY names the capacity reason. NoCapacityError carries the plan's reason. When it is a capacity reason, the worker dispatcher throws DispatchError("NO_CAPACITY", message, { capacityReason }), as it does for LEASE_ID_TAKEN (F25). Otherwise no details, as today. So the socket sends it (F24), and HTTP's 503 body carries capacityReason (F24, F26). A replay answers no details, as today (F32).

5. CLI. A queued line with a reason ends — plus the reason text with only its first letter lower-cased. The text comes from feature Operational metrics kept over hours and shown in the console #623 Shared values "Reason text"; the platform word is the request's (iOS, Android).

6. Console. The inspector gains a "Why" fact after Queue, with the reason text, for an entry with a reason. No row otherwise. A reason the console does not know shows as its raw word, as a stage does (F29).

7. Two text tables, each checked. The CLI keeps its table in src/cli, the console in ui/src/model (F30). Each side has one test that its table covers every reason the contract's enum lists.

8. A second provisioning failure rejects. A first failure still plans again at once, as today (F11). A second no longer enqueues: the waiter is rejected with PROVISION_FAILED, kind domain, HTTP 500, exit code 1 (F38), whose message is the driver's error. A probe (fleetRequestId set) that fails twice is declined with PROVISION_FAILED too, not NO_CAPACITY. The rejection wakes the queue, as the boot-timeout and component-being-removed rejections do (src/leasing/lease-acquisition-coordinator.ts:828-845), so the next waiter plans. Those two stay as they are (decided by the maintainer, 2026-10-18).

Modules touched

Searches: waitingRequestSchema, WaitingRequest, places(, blocked(, NoCapacityError, "NO_CAPACITY", CapacityRefusalReason, formatWaitingRequests, WaitingRow.

• src/contract/schemas.ts, src/contract/errors.ts, src/contract/protocol.ts
• src/core/index.ts (exports CapacityRefusalReason)
• src/leasing/acquisition-planner.ts, src/leasing/lease-acquisition-coordinator.ts, src/leasing/lease-request-book.ts, src/leasing/wait-queue.ts, src/leasing/create-leasing.ts
• src/daemon/dispatcher.ts
• src/cli/index.ts
• ui/src/model/waiting.ts, ui/src/screens/waiting/index.tsx
• e2e/waiting-requests.test.ts: its five line patterns (F36) gain the reason suffix, and the single-host test pins the limits below
• Docs: docs/CLI.md, docs/HTTP-API.md, docs/CONSOLE.md, docs/CLIENT.md; manuals src/leasing/AGENTS.md, src/contract/AGENTS.md

flowchart LR
  cli["CLI (changed)"] --> d["worker dispatcher (changed)"]
  http[HTTP] --> d
  ui["console Waiting (changed)"] --> http
  d --> lease["leasing: coordinator, book (changed)"]
  lease --> plan["planner (changed)"]
  plan --> cap[capacity]
  d --> contract["contract (changed)"]

sequenceDiagram
  participant G as Gateway or client
  participant D as Worker dispatcher
  participant C as Coordinator
  participant P as Planner
  G->>D: lease.request noWait
  D->>C: request
  C->>P: plan
  P-->>C: no-capacity, ram-budget
  C-->>D: NoCapacityError(ram-budget)
  D-->>G: NO_CAPACITY, details.capacityReason ram-budget
  G->>D: list.get requests
  D-->>G: [{ stage: queued, reason: platform-running-limit, ... }]

Contract and event changes

• waitingRequestSchema gains reason?: enum of the seven values in feature Operational metrics kept over hours and shown in the console #623 Shared values "Waiting reason (worker)" (decided by the maintainer, 2026-10-18). A value outside the enum fails the parse, and so does reason on a starting entry.
• ErrorDetailsMap.NO_CAPACITY becomes { capacityReason?: enum of the four }; NO_CAPACITY joins CODES_WITH_DECLARED_DETAILS.
• New error code PROVISION_FAILED: kind domain, httpStatus 500, cliExitCode 1 (F38); no details; feature Operational metrics kept over hours and shown in the console #623 Shared values "PROVISION_FAILED" (decided by the maintainer, 2026-10-18).
• This PR bumps the protocol by one, min with it (F31; feature Operational metrics kept over hours and shown in the console #623 Shared values "NO_CAPACITY detail").
• Events: none.

Other code on the same state

Searches: readers of queue.pending(), queue.list(), .places() outside tests (4 hits in src/leasing, 5 in src/gateway); writers of waiter fields in lease-acquisition-coordinator.ts.

• Warm pool waitingDemand (lease-acquisition-coordinator.ts:621-635) and #heldByOlder (:729-738): read spec and state, not the reason. No conflict.
• cancelPending (:609-619): rejects a queued waiter; it leaves the list (F17), and now wakes the queue (F34). A wake only plans the head and physical waiters under the lock, as every wake does.
• The gateway (F18): shares the book and queue code. Its own entries carry no reason; a worker's entries keep theirs (F35).
• A waiting request says why each worker cannot take it, on a gateway #637: adds workers to the same schema and reads details.capacityReason, and tells a worker's PROVISION_FAILED from NO_CAPACITY (decided by the maintainer, 2026-10-18). A waiting request says why each worker cannot take it, on a gateway #637 depends on this task.
• The event history is a SQLite file, history.db #629, usage.get and simlock stats from the projections #631, A quarantined device says what failed #635 each bump the protocol. The PR that merges second bumps again.
• Overview and Waiting: a range picker, wait p95, leases granted and three charts #633: adds the wait p95 to the Waiting page. Other lines of the same screen; whichever merges second takes the base branch.

Failure modes

• Restart mid-step: the reason is in memory; the restart fails the request (F16). Nothing left behind; no test needed.
• Two callers at once: a listing reads the waiter's reason as the last plan left it. One thread; no torn read; no test needed.

Rules in play

• architecture.md 10: "A rule is enforced in exactly one place." The reason comes from the plan; each text table is checked against the contract's enum by a test; the capacity list is checked by the compiler.
• architecture.md 16: "Policy is a pure function". The planner returns the reason; it stores nothing.
• architecture.md 8: "The daemon owns all state; the CLI is a thin client." The CLI only prints the field.
• safety.md 10: "Anything that arrives over the wire is a claim". reason and capacityReason are closed enums in the contract.
• src/contract/AGENTS.md: "Every change to a contract shape bumps it by one in the same PR."
• testing.md 6, 7: through createLeasing, and the planner's own test for its many cases.
• documentation.md 3: no repo path in what the CLI prints.

Tests

Seam: createLeasing through createTestEngine (src/leasing/testing.ts) and its waitingRequests(), which list.get requests returns (F21); model on src/leasing/create-leasing.test.ts:3757. The planner's cases in src/leasing/acquisition-planner.test.ts.

• Planner: each blocked path returns its reason: not tried by the planner; warm-pool boot waiting-for-device; boot refused ram-budget, global-running-limit, platform-running-limit; provision refused device-limit, ram-budget, global-running-limit, platform-running-limit; victim claim taken at device-limit and at platform-running-limit, with the eviction plans carrying it; physical fits held only for an older request behind-earlier-request; boot claim taken waiting-for-device; physical no-free-physical. Extend "waits or rejects when policy cannot reserve capacity" (acquisition-planner.test.ts:211).
• The same reasons come back on no-capacity under noWait.
• A queued request at the iOS running limit lists reason: "platform-running-limit"; a second one behind it lists behind-earlier-request; the stored request record has no reason.
• The reason follows the latest attempt: when the head is granted, the request behind it becomes head and lists its own new reason, not behind-earlier-request.
• A first provisioning failure still plans again at once, and a success then grants (decided by the maintainer, 2026-10-18).
• A second provisioning failure rejects the request with PROVISION_FAILED and the driver's message; nothing stays listed for it; the request behind it plans again and lists its own reason (decided by the maintainer, 2026-10-18).
• A probe (fleetRequestId set) whose provisioning fails twice answers PROVISION_FAILED, not NO_CAPACITY (decided by the maintainer, 2026-10-18).
• Cancelling the queued head makes the request behind it plan again and list its own reason, not behind-earlier-request.
• A request while the warm pool boots a fitting device lists waiting-for-device.
• A failed eviction lists the capacity reason that asked for it; a failed physical grant lists no-free-physical; a request whose only match is reclaiming lists the planner's capacity reason.
• A waiting physical request with every fit leased lists no-free-physical (physical-leasing.test.ts:582).
• A starting request lists no reason: one being provisioned, and one held at its first plan.
• A gateway's own queued entry lists no reason: extend src/gateway/fleet-coordinator.test.ts:4309. A gateway's list.get requests keeps a worker's own entry's reason: extend src/gateway/dispatcher.test.ts:1436.
• Dispatcher: a noWait lease.request refused at the RAM budget, at the device limit and at the platform running limit rejects NO_CAPACITY with details.capacityReason of each; one behind an earlier request rejects NO_CAPACITY with no details; a replay of the RAM-budget request by its idempotency key rejects NO_CAPACITY with no details. Model on src/daemon/dispatcher.test.ts:473.
• HTTP: POST /v1/lease-requests with noWait at the running limit answers 503 with capacityReason in the error body (src/http/app.test.ts).
• Contract: waitingRequestSchema accepts each of the seven reasons on a queued entry, and refuses "bogus" and any reason on a starting entry; NO_CAPACITY is in CODES_WITH_DECLARED_DETAILS (src/contract/schemas.test.ts, errors.test.ts).
• Contract: PROVISION_FAILED is kind domain, HTTP 500, exit code 1 (errors.test.ts) (decided by the maintainer, 2026-10-18).
• CLI: a queued request prints — <text> for every reason, platform-running-limit once for iOS and once for Android, only the first letter lower-cased; a queued request with no reason and a starting one print as today. Extend src/cli/index.test.ts:6284.
• CLI and console tables: each side's table has a text for every reason the contract's enum lists (one test in src/cli, one in ui/src/model).
• Console model: the chosen row's Why is the text for every reason, the raw word for an unknown one, and absent with no reason (ui/src/model/waiting.test.ts).
• Console screen: the inspector shows a Why fact after Queue with the text, and no Why fact for a starting request (ui/src/screens/waiting/waiting.test.tsx:76).
• E2E, single host (e2e/waiting-requests.test.ts:37-55), limits { ios: { maxDevices: 2, maxRunning: 1 }, maxRunning: 2 }, one lease held: agent-b's line ends — full: running limit for iOS, agent-c's — behind an earlier request.
• E2E, gateway (:57-101): the worker's local-b line ends — full: device limit, on the worker and on the gateway; the fleet line is unchanged.

Done when

• The tests above pass.
• On the fake driver, with ios.maxDevices 2, ios.maxRunning 1, global maxRunning 2 and one iOS lease held, a second simlock lease waits; simlock list --requests prints its line ending — full: running limit for iOS (e2e/waiting-requests.test.ts:37-55), and the console's Waiting inspector shows the same text under Why.
• With the same limits, one iOS lease held and no request waiting, lease.request with noWait: true over the socket is the head, and answers NO_CAPACITY with details.capacityReason: "platform-running-limit".
• docs/CLI.md, docs/HTTP-API.md, docs/CONSOLE.md and docs/CLIENT.md describe the reason and the detail.
• The error tables in docs/CLI.md, docs/HTTP-API.md and docs/CLIENT.md list PROVISION_FAILED (F39) (decided by the maintainer, 2026-10-18).

Out of scope

• A gateway's own entries and per-worker reasons (A waiting request says why each worker cannot take it, on a gateway #637).
• A reason on a starting request.
• Keeping reasons across a restart, or in events.
• Waiting only behind same-platform requests (A request waits only behind requests for the same platform #387).
• The gateway's own NO_CAPACITY carrying a capacity reason.

Depends on

• A quarantined device says what failed #635

Written by an agent.