Skip to content

fix(server): preserve task loaders and checkpoint resume state - #387

Merged
tastelikefeet merged 18 commits into
dev-swift-v5from
fix/server-task-checkpoints-20261116
Nov 17, 2026
Merged

tastelikefeet merged 18 commits into
dev-swift-v5from
fix/server-task-checkpoints-20261116

Conversation

@Yunnglin

@Yunnglin Yunnglin commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

修复服务端任务模型加载、租户任务头与训练状态隔离、断点恢复,以及多模态 LoRA 目标选择和 Megatron 多槽位一致性。

  • 验证:本次定向 CPU 回归 73 项及 4 项子用例通过,另验证双进程 CPU 分片写回与恢复,本次未重跑 PPU;此前两卡 FSDP 任务头训练/恢复、晚绑定及跨槽位隔离验证。此前 PPU 验收覆盖图文 SFT 四组合、LoRA 取消/中断与隔离、LoRA 及 Megatron 全参扩缩容。
  • GRPO 补测:两后端 LoRA 的独立 GSM8K 测试确认输出压缩,Transformers/Megatron 分别减少 32.5%/61.0%;Megatron 正确率下降约 4.1 个百分点,保留该代价。
  • 限制:Transformers 全参从零重建后的冷恢复仍出现 600 秒超时;图文正确率未提升,Transformers 全参有所回退。ST 全参和完整 ST Megatron 尚有缺口,StartLux Transformers 全参未验收;撤回确定性适配后的默认计算路径,PPU 续训数值对齐待复测。

Yunnglin and others added 9 commits November 16, 2026 07:20
Keep task heads, optimizer state and deterministic continuation aligned across PPU training and cold resume.
Validate model families before allocation and carry custom components, task heads and encoded tensors through the HTTP training path.
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Codex <noreply@openai.com>
@Yunnglin
Yunnglin marked this pull request as ready for review November 16, 2026 10:48
Copilot AI balanced review requested due to automatic review settings November 16, 2026 10:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Preserve lazy optimizer state across cold resume, reject incomplete saves before weight collection, and align Megatron checkpoint calls with supported APIs. Remove unverified decision handling and unrelated loader changes while tightening torch configuration validation.
Comment thread src/twinkle/model/megatron/megatron.py Resolved

for module in self.model.modules():
if type(module).__name__ == 'Qwen3_5GatedDeltaNet':
module.chunk_gated_delta_rule = torch_chunk_gated_delta_rule

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个是修复什么的

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已回退相关代码,原本用于断点续训数值对照时切换 GDN 计算实现,但没有独立证据证明它是恢复问题的必要修复。

return super()._task_head_state(adapter_name)

def _restore_task_head_state(self, adapter_name, state) -> None:
head = self._attached_head(adapter_name)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个函数的含义是?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已回退相关代码,任务头沿用已有保存恢复流程

@tastelikefeet
tastelikefeet merged commit 2a75ee1 into dev-swift-v5 Nov 17, 2026
0 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants