Skip to content

Commit cdda80d

Browse files
igalshilmanclaude
andauthored
Sessions: a handler that watches for its cancellation is not suspended on it (#312)
A handler with explicit cancellation watches for its cancellation (`internal.cancelled(ctx)`, which an agent's turn awaits in the background for the whole turn) to stop work it does outside Restate. With nothing else pending, the selector asked the VM for the cancel signal alone: the VM told Restate that the handler awaits it, and once Restate closed the input (its inactivity timeout), suspended the invocation on it. The invocation stayed suspended on its cancel signal, holding its key, until a cancellation. With only the cancellation pending, the selector now asks for it together with the attempt's working run: a run named `work`, made once per attempt, handed out by the VM like any run and never ended while the attempt lasts. While a run executes, the VM does not suspend on a wait the run is part of, and sends no awaiting-on hint; a cancel signal that comes is still found at once. With anything else pending the working run is left out, and the invocation suspends on what it waits for, as before. The selector's `close`, as the handler returns, fails what still waits and ends the working run before the output. The TypeScript side is restatedev/sdk-sessions#26. Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4a440ff commit cdda80d

5 files changed

Lines changed: 384 additions & 36 deletions

File tree

‎docs/agents/turns.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -455,6 +455,14 @@ sequenceDiagram
455455
does nothing), never the message that started after it.
456456
- The turn ends cancelled whenever the cancellation comes, and the next
457457
message starts afresh: the cancelled prompt is over.
458+
- The cancellation reaches the turn's work at once while Restate keeps the
459+
turn's input open. Restate closes it once the turn has sent and received
460+
nothing for the inactivity timeout (a minute by default: a long model
461+
request, or a slow tool). The turn works on, as watching for the
462+
cancellation holds nothing, and a cancellation then reaches it at its next
463+
step, when it next commits or calls. For turns whose steps take longer,
464+
raise Restate's inactivity timeout (the server's
465+
`worker.invoker.inactivity-timeout`).
458466
- A cancellation that comes once the turn answered changes nothing: the
459467
answer stands.
460468
- A tool cut short by the stop raises the stop's reason, so the model never

‎python/restate/agents/_turn.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -371,7 +371,12 @@ async def chat_turn(ctx: AgentContext, text: str, answer: Callable[[AgentContext
371371
While it answers, the object's ``status`` reports the turn. Cancelling the
372372
invocation stops the turn (through ``stop_signal_of``) whenever it comes,
373373
and cancels the calls its tools made; the handler then fails with the
374-
cancellation. If the attempt ends first, the turn stops with
374+
cancellation. The cancellation reaches the turn's work at once while
375+
Restate keeps the turn's input open; once Restate closed it (nothing
376+
sent or received for its inactivity timeout: a long model request, say),
377+
at the turn's next step, when it next commits or calls (see
378+
``cancelled`` of ``restate.sessions.internal``). Watching for it holds
379+
nothing: the turn works on. If the attempt ends first, the turn stops with
375380
``AttemptEnded`` and the next attempt carries on. The turn publishes when
376381
it starts (the companion's ``start`` does), after each write of its steps
377382
that commits (a reply with tool calls, the tools' results: ``answer_once``

‎python/restate/sessions/_object.py‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,11 @@ def _sdk_handler(entry: _Entry) -> Callable[..., Awaitable[Any]]:
298298
@functools.wraps(entry.fn)
299299
async def handler(ctx: ServerInvocationContext, *args: Any) -> Any:
300300
host = _ContextHost(ctx, shared=entry.view)
301-
return await _attempt(entry, host, ctx.request(), args)
301+
try:
302+
return await _attempt(entry, host, ctx.request(), args)
303+
finally:
304+
# Before the SDK writes the output: the working run ends first, in the journal.
305+
host.selector.close()
302306

303307
return handler
304308

@@ -327,6 +331,16 @@ async def stop_when_cancelled():
327331
328332
asyncio.ensure_future(stop_when_cancelled())
329333
334+
It watches for the cancellation, and holds nothing: awaited alone, it
335+
neither suspends the invocation nor tells Restate that the handler waits
336+
(the attempt's journal has a run named ``work`` instead, for the
337+
handler's own work; see ``_selector.py``). A cancellation reaches the
338+
handler at once while Restate keeps the invocation's input open. Once
339+
Restate closed it (the handler sent and received nothing for the
340+
inactivity timeout), a cancellation reaches the handler at its next
341+
await of Restate's, which suspends the invocation and resumes it at once
342+
(in the next attempt), or not at all if it awaits nothing more.
343+
330344
Args:
331345
ctx: The context of a handler with ``explicit_cancellation=True``.
332346

‎python/restate/sessions/_selector.py‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,24 @@
1717
cancellation each take a single unresolved future. So every wait registers
1818
with the selector, which asks the VM for progress on whichever of them
1919
completes first, and which is the only reader of the invocation's input.
20+
21+
To ask is to say that the handler waits, but a handler that watches for its
22+
cancellation does not wait for it: see ``FutureSelector`` for the working
23+
run that the selector asks for with it.
2024
"""
2125

2226
from __future__ import annotations
2327

2428
import asyncio
29+
import contextlib
2530
import logging
2631
from collections.abc import Callable, Mapping
2732
from typing import Any, Protocol
2833

2934
from restate.exceptions import SdkInternalException, SuspendedException, TerminalError
3035
from restate.server_context import DisconnectedException, ServerInvocationContext
3136
from restate.vm import (
37+
CANCEL_HANDLE,
3238
DoProgressAnyCompleted,
3339
DoProgressCancelSignalReceived,
3440
DoProgressExecuteRun,
@@ -69,6 +75,19 @@ def execute_run(self, handle: int) -> None:
6975
"""Start the run of ``handle``, which signals its completion through ``receive``."""
7076
...
7177

78+
def work(self) -> int:
79+
"""Return the handle of the attempt's working run, made at the first call.
80+
81+
It is a run that the VM hands out like any other (``DoProgressExecuteRun``),
82+
and that does not end while the attempt lasts: it stands for the
83+
handler's work outside Restate.
84+
"""
85+
...
86+
87+
def end_work(self) -> None:
88+
"""End the working run, if there is one, as the attempt ends."""
89+
...
90+
7291
def notify_input(self, chunk: Mapping[str, Any]) -> None:
7392
"""Pass a chunk of the request body to the VM."""
7493
...
@@ -82,6 +101,17 @@ def on_end(self) -> None:
82101
...
83102

84103

104+
CANCELLATION: int = CANCEL_HANDLE
105+
"""The handle of the invocation's cancel signal: the VM's, and what ``ctx.cancellation()`` awaits."""
106+
107+
108+
def cancellation_alone(future: UnresolvedFuture) -> bool:
109+
"""Return whether ``future`` waits for the invocation's cancellation, and nothing else."""
110+
if isinstance(future, SingleUnresolvedFuture):
111+
return future.handle == CANCELLATION
112+
return all(cancellation_alone(child) for child in future.children)
113+
114+
85115
def is_ready(future: UnresolvedFuture, completed: Callable[[int], bool]) -> bool:
86116
"""Return whether ``future`` needs no more progress."""
87117
if isinstance(future, SingleUnresolvedFuture):
@@ -101,6 +131,25 @@ class FutureSelector:
101131
reaches all of them. Registering a wait while the loop reads input wakes
102132
the loop, so the new wait is included in the next check. The loop runs as
103133
long as something is pending.
134+
135+
To ask is to say that the handler waits: the VM tells Restate what for,
136+
and suspends the invocation on it once the input closed (Restate's
137+
inactivity timeout). That holds for every wait but one. A handler with
138+
explicit cancellation waits for its cancellation (``cancelled(ctx)``) to
139+
stop work it does outside Restate, a model's request, say: it watches for
140+
it, and does not wait for it. Reported as awaited, the cancel signal
141+
would be all Restate believes the handler waits for; and suspended on it,
142+
the invocation would end its attempt in the middle of that work, with a
143+
cancellation as the only thing to resume it.
144+
145+
So with only the cancellation pending, the loop asks for it together with
146+
the attempt's working run (``SelectorDriver.work``), which says what is
147+
true: the handler is at work on something of its own, as inside
148+
``ctx.run``. While a run executes, the VM reports no await, and does not
149+
suspend on a future the run is part of, whether the input is open or
150+
closed; and it still finds the cancel signal when it comes. With anything
151+
else pending, the working run is left out, and the invocation suspends on
152+
what it waits for, the cancellation among it, as before.
104153
"""
105154

106155
def __init__(self, driver: SelectorDriver, disconnected: Callable[[], BaseException]) -> None:
@@ -140,6 +189,17 @@ async def wait(self, future: UnresolvedFuture) -> None:
140189
finally:
141190
self._waits.pop(waiter, None)
142191

192+
def close(self) -> None:
193+
"""End the attempt as the handler returns: fail what still waits, and end the working run."""
194+
if self._end is None:
195+
self._end = SuspendedException
196+
self._fail_all(self._end)
197+
if self._waiting:
198+
# Its next pass finds nothing pending, and ends the loop.
199+
self._waiting = False
200+
self._driver.wake()
201+
self._driver.end_work()
202+
143203
def _start(self) -> None:
144204
if self._task is None:
145205
self._task = asyncio.create_task(self._loop())
@@ -181,6 +241,9 @@ async def _loop(self) -> None:
181241
pending = [future for waiter, future in self._waits.items() if not waiter.done()]
182242
if not pending:
183243
return
244+
if all(cancellation_alone(future) for future in pending):
245+
# The handler watches, and waits for nothing: it works (see the class).
246+
pending.append(SingleUnresolvedFuture(driver.work()))
184247
await driver.flush_output()
185248
response = driver.do_progress(
186249
pending[0] if len(pending) == 1 else FirstCompletedUnresolvedFuture(pending)
@@ -252,6 +315,8 @@ class _ContextDriver:
252315
def __init__(self, context: ServerInvocationContext, on_end: Callable[[], None]) -> None:
253316
self.context: ServerInvocationContext = context
254317
self.on_end: Callable[[], None] = on_end
318+
# The working run's handle, once made.
319+
self._work: int | None = None
255320

256321
def do_progress(self, future: UnresolvedFuture) -> Any:
257322
return self.context.vm.do_progress(future)
@@ -271,6 +336,9 @@ def wake(self) -> None:
271336
self.track(asyncio.ensure_future(event))
272337

273338
def execute_run(self, handle: int) -> None:
339+
if handle == self._work:
340+
# The working run has no action: it executes until the attempt ends.
341+
return
274342
context = self.context
275343
fn = context.run_coros_to_execute.pop(handle)
276344

@@ -290,6 +358,21 @@ def notify_input(self, chunk: Mapping[str, Any]) -> None:
290358
if not chunk.get("more_body", False):
291359
self.context.vm.notify_input_closed()
292360

361+
def work(self) -> int:
362+
if self._work is None:
363+
# Journaled as a run named ``work``, whose action is the handler's own work.
364+
self._work = self.context.vm.sys_run("work").handle
365+
return self._work
366+
367+
def end_work(self) -> None:
368+
handle, self._work = self._work, None
369+
if handle is None:
370+
return
371+
# The work is over: so is its run, in the journal of an invocation that goes on to answer. A VM that a
372+
# suspension or an error closed refuses: the next attempt, or nothing, ends the run.
373+
with contextlib.suppress(Exception):
374+
self.context.vm.propose_run_completion_success(handle, b"")
375+
293376
def track(self, task: asyncio.Future[Any]) -> None:
294377
self.context.tasks.add(task)
295378

0 commit comments

Comments
 (0)