Repository navigation
feat(eot): add agent context, the mini model, and turn length - #8113
Open
chenghao-mou wants to merge 16 commits into
Open
chenghao-mou wants to merge 16 commits into
chenghao-mou wants to merge 16 commits into
Conversation
Send the last assistant message through EotInputChatContext before each cloud prediction. Clear prior text when the chat context has no assistant text. Addresses AGT-3730
chenghao-mou
added
turn-detector
6 of 8 tasks
livekit-local-inference 0.2.8 ships v1.1-mini, which reads the last 2.0 s of audio and takes the turn length, counted from the user's first speech, as a feature. The local transport now buffers EOT_MAX_SAMPLES and passes turn_samples; AudioRecognition marks the turn start on VAD start of speech. The per-language thresholds follow the int8 refit rule used for the cloud model: on eot-bench, under the default audio-EOT endpointing (0.3 s, 2.5 s), the fp32 model's best latency at a 10% cutoff budget sets each language's target, and the shipped quantized model takes the fastest threshold whose cutoff rate stays within the fp32 one.
Rename mark_speech_start(lookback) to on_start_of_turn(started_at), after the endpointing peer on_start_of_speech(started_at). AudioRecognition now calls it once per turn, where it sets _user_turn_start, so the transport no longer filters repeated starts and a turn that starts from STT gets a start too. The stream turns the start time into a duration when called, so the channel delay can't stretch the turn.
Both on_start_of_turn methods now take started_at; the local transport turns it into a sample position when it reads the sentinel. The drain lag this adds is sub-millisecond on an idle loop, and a long stall already delays inference.
chenghao-mou
force-pushed
the
chenghao/feat/AGT-3736-mini-2s-eot-model
branch
from
October 27, 2026 10:34
cf289f1 to
95b87ba
Compare
The local transport read the clock when it drained the start-of-turn sentinel. During a cloud reconnect the audio channel backs up, so after a fallback to local the turn length was too long by the backlog. The stream now converts the start time to samples when it queues the sentinel, and the transport takes that sample count. - Keep one turn counter in the local transport, and send at least the audio the model reads. - Tell the turn detector only a known speech start. A turn opened at end of speech keeps the count from the last flush. - Correct the memory figure to the measured ~450 MB.
chenghao-mou
force-pushed
the
chenghao/feat/AGT-3736-mini-2s-eot-model
branch
from
October 27, 2026 12:01
2d7017d to
c12b0b5
Compare
The sentinel carries the elapsed time measured when it is queued, and the local transport converts it to samples. The name no longer collides with the turn_samples passed to predict, which counts up to inference.
The cloud end-of-turn model reads the turn duration too. The cloud transport counts the turn the way the local transport does, from the turn start since the last flush, and sends it as turn_samples on inference_start (livekit-protocol 1.1.31). The gateway adds the audio that reaches it later. Addresses AGT-3872
…-context' into chenghao/feat/AGT-3736-mini-2s-eot-model # Conflicts: # livekit-agents/livekit/agents/inference/eot/transports.py
…6-mini-2s-eot-model # Conflicts: # livekit-agents/pyproject.toml
chenghao-mou
changed the title
predict() called the transport directly, so frames captured before it but still queued went out after inference_start: the turn length left them out, and the gateway had to wait for later audio to collect them. predict() now queues the request behind the audio, as flush and the turn start already are. The transport runs it after every earlier frame, so turn_samples covers all audio captured before the call, and a request replaced before it runs is not sent. The request carries a snapshot of the chat context items. Addresses AGT-3872
chenghao-mou
marked this pull request as ready for review
November 5, 2026 13:34
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Addresses AGT-3736
Addresses AGT-3736
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This combines #7787 into the end-of-turn changes for v1 (cloud) and v1-mini (local), so they ship and get reviewed together.
Agent context (v1): before each cloud prediction, the stream sends the last assistant message as
EotInputChatContext. Empty context clears prior text. Local inference stays audio-only.Mini model (v1-mini): the next local model (
livekit-local-inference0.2.8) reads the last 2.0 s of audio and takes the turn length as a feature. The per-language thresholds are refit for it on eot-bench.Turn length (both):
AudioRecognitionreports the user turn start once per turn throughon_start_of_turn. The local transport passes the turn length to the model. The cloud transport sends it asturn_samplesoninference_start(livekit-protocol1.1.31, livekit/protocol#1937).predict()now queues the request behind the audio, as flush and the turn start already are, so the count covers every frame captured before the call and the gateway needs no wait. A request replaced or cancelled before it reaches the transport is not sent.Companions: livekit/agent-gateway#1571 forwards both fields, and livekit/infserve#145 serves the model that reads them.
Warning
Blocked on the
livekit-local-inference0.2.8 release (livekit/agents-private#551).uv.lockstill resolves 0.2.7, so CI fails until it ships.Addresses AGT-3730
Addresses AGT-3736
Addresses AGT-3872
Initial prompt and agent context
Model: Claude Opus 5.5
Then, on the thresholds:
The 10% cutoff budget under the default endpointing delays was chosen as the anchor. Then: "yes, open the PR and open the node js PR too."
Later, adding the turn length to the cloud path (AGT-3872):
Later, combining the client PRs:
Then, on the snapshot point: