The warm pool learns what to keep warm, with no configuration #771

Request: none

Problem

A worker boots nothing ahead of demand unless an operator writes warmPool.targets into the config file and restarts the daemon. Nobody does, so the first lease of every model, OS and mode on a worker pays a cold boot (about 60 s for an iOS simulator, about 160 s slim, about 70 s for an Android emulator). After a lease ends, the pool reboots the device only if the lease ended less than idle.shutdownAfterMs ago and a slot is free, one boot at a time; the next request often arrives before that lands, and a restart starts cold. A queued request gets a boot only when it is the queue head and a shut-down device already matches; a burst of five agents asking for one kind pays five boots one after the other. The pool learns nothing from what was leased.

Two reliability holes found on 2026-10-20 sit beside this: a keep boot the RAM budget refuses is declined silently on every pass, and a target is never reported short while any waiter sits on its platform. Two more findings outside the pool were filed on their own (see Non-goals).

Who it is for

The maintainer of a worker that serves parallel coding agents, who installs Simlock and never edits its config. The agents that lease from it, which today wait a cold boot on every new kind and on every burst.

Outcome

With no configuration, a worker keeps what it has leased: one device per kind it leased recently, as many as that kind was held at once in the last 24 hours counting the ones leased right now, newest kinds first within the capacity budget, booted back when released, and booted again after a restart. Every queued request gets a device started for it ahead of its turn, several at once. A kind the worker cannot keep says why in status. Explicit warmPool.targets stay as an override and are never displaced. simlock nuke forgets what was learned.

How it works for the user

sequenceDiagram
  participant A as agent
  participant W as worker
  participant P as warm pool
  A->>W: lease iPhone 16 / 18.4
  W-->>A: granted, source provisioned (60 s)
  W->>P: lease.granted: kind learned
  A->>W: release
  W->>P: lease ended; reclaim erases, shuts down
  P->>P: pass: kind wants 1, has 0: boot it back
  A->>W: lease iPhone 16 / 18.4 (later, or after a restart)
  W-->>A: granted, source warm (under 1 s)
  A->>W: 5 leases of one kind at once
  P->>P: pass: one boot or creation per queued request, up to the boot cap
  W-->>A: each granted as its device is ready

# a fresh worker, no config.json
$ simlock lease --platform ios --device "iPhone 16" --os 18.4 --agent-id a1
{"lease":{"id":"..."},"device":{...},"timing":{"estimatedBootMs":60000,...},"source":"provisioned"}
$ simlock release <id>
$ simlock status
warm pool: enabled, reserve ios 0 android 0
  iPhone 16 / 18.4 / full   learned   wanted 1  ready 0  booting 1
$ simlock daemon stop; xcrun simctl --set ~/.simlock/devices/ios shutdown all; simlock status   # a minute later
warm pool: enabled, reserve ios 0 android 0
  iPhone 16 / 18.4 / full   learned   wanted 1  ready 1  booting 0
$ simlock lease --platform ios --device "iPhone 16" --os 18.4 --agent-id a2
{..., "source":"warm"}
$ simlock nuke --yes
$ simlock status
warm pool: enabled, reserve ios 0 android 0

Today the same session shows no warm pool line after the release, source: booted or provisioned on the second lease, and nothing ready after a restart.

Examples

What the pool keeps on a worker with two iOS running slots and no warmPool.targets:

Five requests for one kind queued at once on a 12-CPU worker with one ready device:

What could go wrong

• Devices boot back that nobody asked for yet, holding RAM on a quiet machine: by design, within the same running limits as today; nothing extra is made while a burst is held.
• Two kinds that alternate under a budget of one trade places, one boot each time: the LRU rule the maintainer chose.
• A kind keeps failing to boot: it is tried three times in a row, then forgotten with a warning, not forever.
• A request in a burst is granted a device the pool created rather than one it started itself: the same device either way; the rare race where both create leaves one idle device the idle rules remove.

Words used

• kind: a virtual device's exact spec: platform, model, OS version, mode (slim or full) and image tag, e.g. iPhone 16 / 18.4 / full.
• learned kind: a kind this worker granted a lease of, kept ready like a target; shown with origin learned.
• configured target: a warmPool.targets entry; origin configured. A kind that is both is one entry, origin configured, with the larger count.
• count: how many devices of a kind the pool wants, ready or leased: the most leases of it held at the same moment in the last 24 hours, at least one. A configured target's count is ready devices only, as today.
• budget: the running slots, device limit, RAM budget and boot cap the pool may fill, as today (ADR 0017 §4).
• boot cap: warmPool.maxConcurrentBoots, the most pool boots and creations running at once.
• displaced: an older learned kind's idle device shut down so a newer learned kind can take its slot.
• on its way: a device being created or booted for nobody in particular, which a request that fits waits for instead of starting another.
• forgotten: a learned kind removed from the record; its next lease learns it again.

Non-goals

• Fleet-level targets or learning on the gateway. Each worker learns its own demand; the gateway keeps routing by warm hits.
• Any change to lease.identity: a learned kind under fresh pre-creates its replacement as a configured target does.
• Changing configured targets without a restart.
• A new config key: warmPool.enabled stays the one switch.
• Android's quoted boot time on a snapshot restore, and shut-down devices counting against the RAM budget: filed as Android quotes a 70 s boot for a 4 s snapshot restore #778 and A shut-down device still counts against the RAM budget #779.
• A creation for the queue head: leasing's own, as today.

Completion conditions

• With no config, after a lease of a kind ends, the next lease of that kind on that worker is granted source: warm while the budget allows.
• Three leases of one kind held at once make nothing extra while held, and once released leave three devices of it ready while the budget allows.
• After simlock daemon stop and a start, the kinds last leased are ready again before any lease.
• Five requests for one kind queued at once start maxConcurrentBoots boots together, and each request behind the head is granted the device started for it.
• warmPool.maxConcurrentBoots defaults to max(1, floor(cpu / 4)), as simlock config shows.
• simlock status and the console list each learned kind with origin learned, wanted, ready, booting, short and its image tag when it has one; a configured target shows origin configured.
• simlock nuke clears the learned record: nothing is booted back after it until the next lease.
• A learned kind whose runtime was removed disappears from status without a restart.
• A boot the RAM budget refuses is not proposed again every pass.

Open questions

Decisions

• ADR 0038 — The warm pool learns its kinds from leases, and boots for every queued request (PR docs(adr): 0038 the warm pool learns its kinds from leases, and boots for every queued request #772): what a kind is, the 24-hour peak, newest-first filling and displacement, forgetting, the record as the pool's own state.db tables, queue work for every waiter, the boot cap's default, and what status shows.
• ADR 0036 — Daemon state is a SQLite database, and each owner keeps its own tables (PR docs(adr): 0036 daemon state is a SQLite database, and each owner keeps its own tables #755, Daemon state stores in SQLite (state.db) #754): why the record is a table and not a JSON key.

Shared values

Tasks

flowchart LR
  T761["#761 state.db (feature #754)"] --> B["#776 learned kinds"]
  A["#774 queue boots and the boot cap"] --> B
  B --> C["#777 status, doctor, console, docs"]
  ADR["ADR 0038, PR #772"] -.-> A

• The warm pool boots or creates one device per queued request, and the boot cap defaults from the CPU #774 The warm pool boots or creates one device per queued request, and the boot cap defaults from the CPU — a burst of requests starts several boots at once, and a request behind the head is granted the device started for it. Risk: two devices created for one request, or more boots at once than the machine takes.
• The warm pool learns the kinds it leases and keeps them ready, across restarts #776 The warm pool learns the kinds it leases and keeps them ready, across restarts — a worker with no config keeps what it leased ready, replenishes it, reboots it after a restart, forgets it on nuke. Risk: devices booting that nobody asked for, or two kinds trading places in a loop.
• Status, doctor, the console and the docs show learned kinds with their origin #777 Status, doctor, the console and the docs show learned kinds with their origin — an operator sees what holds the worker's RAM and why a kind is short. Risk: a status block that hides learned kinds.

Completion conditions and the task whose Done when delivers them:

Open work in the same area:

• Daemon state stores in SQLite (state.db) #754 state.db, tasks state.db opens and checks at start, and the worker's registry lives in it #761 (the state module, the registry's tables), iOS and Android device data in their own state.db tables #757, Tokens live in state.db on both daemon kinds #758, The gateway's drained workers live in state.db #759, The CLI and MCP show why an auto-started daemon did not start #760: The warm pool learns the kinds it leases and keeps them ready, across restarts #776 adds an owner to the module state.db opens and checks at start, and the worker's registry lives in it #761 builds, in the same worker owner list Tokens live in state.db on both daemon kinds #758 adds tokensOwner to. Depends on: The warm pool learns the kinds it leases and keeps them ready, across restarts #776 on state.db opens and checks at start, and the worker's registry lives in it #761 and Tokens live in state.db on both daemon kinds #758.
• Console: Fleet table and a page per worker #766 (task:ready, feature feat(website): the hero plays one machine, then a fleet of them #763's console) renames ui/src/screens/workers/ and rewrites the worker page's Warm pool tab: the lines Status, doctor, the console and the docs show learned kinds with their origin #777 edits. Depends on: Status, doctor, the console and the docs show learned kinds with their origin #777 on Console: Fleet table and a page per worker #766.
• Events history pages backwards: GET /v1/events with before and limit #764 (task:ready) bumps the protocol: contract shape changes overlap (toolchain.md). Depends on: Status, doctor, the console and the docs show learned kinds with their origin #777 on Events history pages backwards: GET /v1/events with before and limit #764.
• A request waits only behind requests for the same platform #387 (request) one waiting line per platform: The warm pool boots or creates one device per queued request, and the boot cap defaults from the CPU #774's per-request boots lessen the stall it describes; no dependency.
• Lease an Android device that runs as a low-RAM device #484 (request) low-RAM Android devices: a learned kind would carry whatever spec such a device has; no dependency.
• Named baselines: a lease starts from a device the operator prepared #327 (feature:spec) named baselines: changes what a reclaim restores, not what is kept warm; no dependency.
• PR docs(adr): 0037 the console acts through the HTTP routes, on Base UI #762 (ADR 0037, console on Base UI): the ADR behind Console: Fleet table and a page per worker #766; covered by that dependency.

Written by an agent.