Repository navigation
Conversation
This was referenced Nov 12, 2026
sufubao
force-pushed
the
refactor/attention-04-varlen
branch
from
November 13, 2026 00:03
ea29556 to
505f315
Compare
sufubao
force-pushed
the
refactor/attention-05-runtime
branch
from
November 13, 2026 00:03
51c58a9 to
1fbf1fb
Compare
sufubao
force-pushed
the
refactor/attention-04-varlen
branch
from
November 13, 2026 01:55
505f315 to
3105509
Compare
sufubao
force-pushed
the
refactor/attention-05-runtime
branch
2 times, most recently
from
November 13, 2026 03:43
4211a05 to
6576639
Compare
sufubao
force-pushed
the
refactor/attention-04-varlen
branch
from
November 13, 2026 03:43
3105509 to
d3c36c4
Compare
sufubao
force-pushed
the
refactor/attention-04-varlen
branch
from
November 13, 2026 03:55
d3c36c4 to
9678c8d
Compare
sufubao
force-pushed
the
refactor/attention-05-runtime
branch
from
November 13, 2026 03:56
6576639 to
9253195
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
接入模型 LayerAttentionBinding、BatchBuilder、State 与 CUDA Graph 生命周期,删除旧 backend/过渡 wrapper。AttentionRuntime 管理 context、workspace 和 prepare/update/finish/replay;State 保持存储、恢复和释放的所有权。
模型层直接调用 binding.forward(infer_state, inputs, out=...),Binding 内查找当前 Context;indexer 保留 .context(),backend 仍接收具体 Context,prepare/update 仍由 Runtime 管理。中英文模型接入文档同步;回归检查执行 Context 切换、out 传递和错误 owner/phase 拒绝。
模型字段改为 attention_requirements,spec 在 backend 构造前生效。MLA scale 统一进入 spec,参与分组及 planning;移除 Params 的副本。Replay GDN decode 与其他路径统一返回 GDNOutput。Graph wrapper 继续从模型投影维度构造输出,旧 tuning import 和 autotune fixture 的修复保留。
本机 H200,固定 torch2.11+cu130、sglang-kernel0.4.5、FlashInfer0.6.18:最终源码的 attention/Varlen/Graph/GLM/SenseNova/DSV4/speculative 组合测试889项通过,双GPU无跳过。包含非因果 prefill、MLA 自定义 scale、错误输出缓冲区和状态恢复。五步累计固定 Black/Flake8 通过。
Qwen3.5-0.8B BF16 TP1DP1,native/replay MTP1 各64次主模型与draft logits恢复比较最大误差0,prefill/decode Graph和allocator回收通过;GPU/CPU cache、DP copy、PD transport均为本机buffer路径。跨节点PD、完整多模态模型和性能未验收。旧 native MTP test_mtp_verify_path[1] 的精确相等已知失败未放宽,仍保留此前对照证据。
由已关闭的 #1700 拆分,本 PR 为第5/5步,依赖顺序 #1717 → #1718 → #1719 → #1720 → #1721。全部保持 Draft;按序合入,前序进入 main 后再 retarget 后续 PR。CI 仅监听 base=main,第2–5步本地检查不等于 GitHub CI。
本轮实验记录:
/data6/exp_logs/runs,最终组合261113-115108,模型261113-115240;源文件哈希、命令、结果和失败记录均保留。