Skip to content

[5/5] refactor(attention): integrate model bindings and state lifecycle - #1721

Draft
sufubao wants to merge 5 commits into
mainfrom
refactor/attention-05-runtime
Draft

sufubao wants to merge 5 commits into
mainfrom
refactor/attention-05-runtime

Conversation

@sufubao

@sufubao sufubao commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

接入模型 LayerAttentionBinding、BatchBuilder、State 与 CUDA Graph 生命周期,删除旧 backend/过渡 wrapper。AttentionRuntime 管理 context、workspace 和 prepare/update/finish/replay;State 保持存储、恢复和释放的所有权。

模型层直接调用 binding.forward(infer_state, inputs, out=...),Binding 内查找当前 Context;indexer 保留 .context(),backend 仍接收具体 Context,prepare/update 仍由 Runtime 管理。中英文模型接入文档同步;回归检查执行 Context 切换、out 传递和错误 owner/phase 拒绝。

模型字段改为 attention_requirements,spec 在 backend 构造前生效。MLA scale 统一进入 spec,参与分组及 planning;移除 Params 的副本。Replay GDN decode 与其他路径统一返回 GDNOutput。Graph wrapper 继续从模型投影维度构造输出,旧 tuning import 和 autotune fixture 的修复保留。

本机 H200,固定 torch2.11+cu130、sglang-kernel0.4.5、FlashInfer0.6.18:最终源码的 attention/Varlen/Graph/GLM/SenseNova/DSV4/speculative 组合测试889项通过,双GPU无跳过。包含非因果 prefill、MLA 自定义 scale、错误输出缓冲区和状态恢复。五步累计固定 Black/Flake8 通过。

Qwen3.5-0.8B BF16 TP1DP1,native/replay MTP1 各64次主模型与draft logits恢复比较最大误差0,prefill/decode Graph和allocator回收通过;GPU/CPU cache、DP copy、PD transport均为本机buffer路径。跨节点PD、完整多模态模型和性能未验收。旧 native MTP test_mtp_verify_path[1] 的精确相等已知失败未放宽,仍保留此前对照证据。

由已关闭的 #1700 拆分,本 PR 为第5/5步,依赖顺序 #1717 → #1718 → #1719 → #1720 → #1721。全部保持 Draft;按序合入,前序进入 main 后再 retarget 后续 PR。CI 仅监听 base=main,第2–5步本地检查不等于 GitHub CI。

本轮实验记录:/data6/exp_logs/runs,最终组合 261113-115108,模型 261113-115240;源文件哈希、命令、结果和失败记录均保留。

@sufubao
sufubao force-pushed the refactor/attention-04-varlen branch from d3c36c4 to 9678c8d Compare November 13, 2026 03:55
@sufubao
sufubao force-pushed the refactor/attention-05-runtime branch from 6576639 to 9253195 Compare November 13, 2026 03:56
@sufubao
sufubao changed the base branch from refactor/attention-04-varlen to main November 13, 2026 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant