Repository navigation
Conversation
1 of 4 tasks
forward_backward and forward_only now reorder each slice_dp batch on the driver (Ray mode) so every DP rank gets a similar workload, reorder every batch-aligned argument with it, and return the outputs in the caller's order. TransformersModel(dp_balance=False) turns it off. - remote_function(dp_balance=True) asks the model's _dp_balance_call for a plan; calls with object refs, raw logits, unencoded inputs or lists of another length are dispatched unchanged. - Loss.dp_balance_group_size keeps paired samples together (preference and reward losses: 2, REAL: num_generations); embedding and listwise reranker losses, and plain callables, opt out. - calculate_loss rejects per-sample lists after a reordered forward_only. - dp_balance/* metrics are reported by calculate_metric. - The GRPO cookbook no longer calls plan_dp_balance itself. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Skip the blocking _dp_balance_spec round trip once a model reports dp_balance off; the flag is fixed at construction. - Do not reorder when the loss for the call cannot be resolved. - Leave the hidden-size default to estimate_workload. - Drop the unread wrapper._dp_balance and the one-line micro-batch config helper. - Document dp_balance in the TransformersModel docs (en/zh). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Each rank holds only its own slice, so the ranks all-gather their sequence lengths, plan the joined batch as the Ray driver would, swap samples when the plan reorders, and send per-sample outputs back to their rank. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…nment Skip balancing for rank-local token means, batch-dependent objectives and undeclared losses. Keep audited sample-mean and globally normalized sum configurations enabled. Reuse aligned loss signals from reordered forward_only calls, reject fresh nested batch arguments and loss-key changes, and fail explicitly when micro-batch caches cannot represent the full batch. Add numerical gradient and cached-call regression coverage and update both language docs.
Move common configuration and loss/backend specification to TrainableModel. Enable inherited dispatch for model entry overrides, preserve replica scheduling, estimate Megatron fixed micro-batch and CP/SP layouts, and resolve only preplanned ObjectRef shards in workers. Keep unsafe losses, opaque references and unsupported schedules unchanged. Restore local Megatron micro-batch outputs, prevent nested rebalancing, document the shared contract and add Ray/Gloo and backend-layout regression coverage.
lincq2000
force-pushed
the
feat/dp-balance-default
branch
from
November 12, 2026 09:35
dd84174 to
4ec848f
Compare
GRPO and PPO sequence-mean now keep DP balancing with per-token and per-sequence controls (dual-clip, overlong filtering, rollout importance sampling, off-policy sequence masking). Entropy-quantile filtering and FIPO depend on the local batch and still keep their order. A reordered forward_only whose rank batch runs as a single in-order micro-batch now caches the aligned loss arguments, so a following calculate_loss works as in the unbatched path. Add gradient-invariance tests for KTO, pointwise reranker and the GRPO controls. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Resolve conflicts in infra/dispatch.py and model/base.py: - dispatch.py: re-apply dp_balance on top of the _run_here/_call_replicas refactor (sticky/exclusive/control/replay/worker_selector were removed upstream). - base.py: keep both the DP balance helpers and the new response-frame / rollout-feature helpers on TrainableModel. Follow-ups needed for the merged tree to import and test: - base.py: import get_task_spec from twinkle.processor.tasks (twinkle.model.tasks no longer defines it). - collectors: collect_tensor_dict was renamed to tensor_dict. - test_dp_balance_safety: give the test harnesses the new _rollout_features / resolve_output_frame_request helpers. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Resolve conflicts in infra/dispatch.py and tests/model/test_micro_batch.py: - dispatch.py: local mode now runs through _run_here; the local DP balance hook wraps that entry instead of calling the method body directly. - test_micro_batch.py: keep the upstream harness collaborators together with the attribute-capable optimizer group the cached-signal path needs. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR type
PR 说明
关联 PR: #355。将 DP 负载均衡内化到
TrainableModel的公共生命周期和调度路径,默认开启。训练循环只需提交原始 batch,无需自行调用 planner,也无需手动重排逐样本训练字段。整体方案
TrainableModel负责开关、后端与 loss 的适用条件查询、均衡规划及 local 模式下的样本交换。公共slice_dp调度钩子覆盖forward_only/forward_backward,包括子类覆盖的入口;嵌套调用只规划一次,避免重复均衡。TransformersModel、MegatronModel及其 multi-LoRA 变体默认dp_balance=True,设为False保持原始切分。独立构造函数初始化同一套基类状态;未声明支持布局的后端自动跳过。variable_seq_lengths对应的 padding 和 CP / SP 长度对齐,再估算各 DP 副本的总计算量,用于选择样本分配方案。保持流水线的 micro-batch 数和调度不变;Megatron 当前仅对等样本数、完整且不打散样本组的固定 micro-batch 布局启用均衡。动态 batching、存在不完整尾部 micro-batch,或 micro-batch 大小无法被样本组大小整除时自动跳过。input_lengths,控制端只用元数据规划和切分引用,不拉回特征;worker 只解析自己的分片,避免再次切分。无法确认对齐关系的整批引用、缺少长度元数据的远端输入自动跳过。local 模式先汇总长度,仅在需要重排时交换样本并将输出送回原 rank;Megatron 的 micro-batch logps 会在还原前合并,不修改 worker 内的指标缓存。seq-mean-token-mean),包括 dual-clip、overlong 过滤、rollout 重要性采样、off-policy 序列掩码等逐 token / 逐序列控制;CEreduction='sum'(含 chunked / Liger fused);BNPO global;REAL(beta=0);DPO(无 SFT 辅助项)、SimPO、KTO、Reward、Pointwise reranker。forward_only重排后,会缓存已对齐的 loss 参数,供使用相同loss_key的calculate_loss复用,并递归拒绝新传入的 batch 参数。开启 micro-batch 但整个 rank batch 只切出一个 micro-batch 时(如评估路径),同样缓存完整参数。切成多个 micro-batch 的forward_only仅缓存最后一个 micro-batch,重排后禁止单独调用calculate_loss,完整训练应使用forward_backward。model.calculate_metric返回,前缀为dp_balance/*。更新中英文文档及 GRPO cookbook,使用模型层统一接口。Swift 接入与合并顺序
请先合并本 PR,再合并 modelscope/ms-swift#10502。
Swift 配套 PR 将默认开启的
--dp_balance传给各后端构造函数,保留异步样本的长度元数据,并移除 modelscope/ms-swift#10517 / modelscope/ms-swift#10546 中 recipe 层的 planner、手动重排和独立的balance_batch开关。异步兼容能力通过公共的引用与元数据调度接口保留,无需单独修复。验证
以下相关测试共 292 项通过:
tests/model/test_dp_balance_backends.pytests/model/test_dp_balance.pytests/model/test_dp_balance_safety.pytests/model/test_micro_batch.pytests/model/test_grpo_micro_batch.pytests/model/test_batch_context_loss.pytests/infra覆盖梯度不变性与自动跳过回归(含 KTO、Pointwise reranker 及 GRPO / PPO 各控制项)、真实 Ray ObjectRef 调度且控制端不拉取特征、输出顺序还原、单 micro-batch
forward_only后的calculate_loss、TP / PP / CP 副本分片、与实际 Megatron processor 布局对照的代价估算、嵌套调度,以及两进程 Gloo 下 Megatron 风格 micro-batch 输出的交换与还原。git diff --check通过;相较 rebase 后的分支基线,无新增 Ruff 诊断。实验计划
环境:4 × 昇腾 910B(2 卡训练,2 卡 vLLM),Python 3.12,torch 2.10.0 / torch_npu 2.10.0.post4,transformers 5.14.1,vLLM 0.31.0 / vllm-ascend 0.19.1rc2,megatron-core 0.18.2 / MindSpeed 26.2.0,flash-linear-attention 0.5.2,ray 2.59.0。
配置:Qwen3.5-4B GRPO,GSM8K,LoRA(r=16,alpha=32,all-linear),bf16,DP=2。每轮 rollout 8 条 prompt × 8 个采样,每个优化步 32 样本;最大生成长度 2048。FSDP2 使用动态 batching(每个 micro-batch 8192 token);Megatron 使用固定 micro-batch(每 rank 4 行)。同一组训练超参、同一随机种子,仅改变
dp_balance。