Skip to content

feat(model): move DP balancing into TrainableModel, enabled by default - #367

Open
lincq2000 wants to merge 8 commits into
modelscope:dev-swift-v5from
lincq2000:feat/dp-balance-default
Open

lincq2000 wants to merge 8 commits into
modelscope:dev-swift-v5from
lincq2000:feat/dp-balance-default

Conversation

@lincq2000

@lincq2000 lincq2000 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR 说明

关联 PR: #355。将 DP 负载均衡内化到 TrainableModel 的公共生命周期和调度路径,默认开启。训练循环只需提交原始 batch,无需自行调用 planner,也无需手动重排逐样本训练字段。

整体方案

  • 基类统一管理。 TrainableModel 负责开关、后端与 loss 的适用条件查询、均衡规划及 local 模式下的样本交换。公共 slice_dp 调度钩子覆盖 forward_only / forward_backward,包括子类覆盖的入口;嵌套调用只规划一次,避免重复均衡。
  • 各后端共用接口。 TransformersModel、MegatronModel 及其 multi-LoRA 变体默认 dp_balance=True,设为 False 保持原始切分。独立构造函数初始化同一套基类状态;未声明支持布局的后端自动跳过。
  • 跨 DP 副本分配样本。 复用现有的等样本数 KK / snake planner,评估原始分配与候选分配,保持每个 DP 副本的样本数不变,同步重排所有对齐的训练参数,并在收集后将逐样本输出还原到调用方顺序。同一 DP 副本中的 TP / PP / CP peers 接收相同分片。
  • 按后端实际执行布局估算代价。 Transformers 复用固定/动态 micro-batch planner;Megatron 侧先模拟实际的固定 micro-batch 切分、variable_seq_lengths 对应的 padding 和 CP / SP 长度对齐,再估算各 DP 副本的总计算量,用于选择样本分配方案。保持流水线的 micro-batch 数和调度不变;Megatron 当前仅对等样本数、完整且不打散样本组的固定 micro-batch 布局启用均衡。动态 batching、存在不完整尾部 micro-batch,或 micro-batch 大小无法被样本组大小整除时自动跳过。
  • 统一 Ray 与 local 路径。 Ray 调用可在逐样本 ObjectRef 列表旁传入 input_lengths,控制端只用元数据规划和切分引用,不拉回特征;worker 只解析自己的分片,避免再次切分。无法确认对齐关系的整批引用、缺少长度元数据的远端输入自动跳过。local 模式先汇总长度,仅在需要重排时交换样本并将输出送回原 rank;Megatron 的 micro-batch logps 会在还原前合并,不修改 worker 内的指标缓存。
  • 保持 loss 语义。 仅允许已确认梯度权重满足重排不变性的 loss 配置参与,保留成对/分组约束:
    • 参与均衡:GRPO、PPO 策略 loss(seq-mean-token-mean),包括 dual-clip、overlong 过滤、rollout 重要性采样、off-policy 序列掩码等逐 token / 逐序列控制;CE reduction='sum'(含 chunked / Liger fused);BNPO global;REAL(beta=0);DPO(无 SFT 辅助项)、SimPO、KTO、Reward、Pointwise reranker。
    • 保持原顺序:按 rank 局部 token 求均值或依赖本 rank batch 统计的配置,重排会改变梯度。包括 PPO value、GKD / OPSD / MOPD、entropy 分位数过滤、FIPO、SDAR、带 SFT 辅助项的 DPO、CE token-mean、InfoNCE / listwise reranker,以及未声明支持的自定义 loss。
    • DP=1、rank 样本数不齐及返回原始 logits 等情况也会跳过。
  • 保护分步计算接口。 Transformers 的 forward_only 重排后,会缓存已对齐的 loss 参数,供使用相同 loss_key 的 calculate_loss 复用,并递归拒绝新传入的 batch 参数。开启 micro-batch 但整个 rank batch 只切出一个 micro-batch 时(如评估路径),同样缓存完整参数。切成多个 micro-batch 的 forward_only 仅缓存最后一个 micro-batch,重排后禁止单独调用 calculate_loss,完整训练应使用 forward_backward。
  • 统一指标与文档。 均衡指标通过 model.calculate_metric 返回,前缀为 dp_balance/*。更新中英文文档及 GRPO cookbook,使用模型层统一接口。

Swift 接入与合并顺序

请先合并本 PR,再合并 modelscope/ms-swift#10502。

Swift 配套 PR 将默认开启的 --dp_balance 传给各后端构造函数,保留异步样本的长度元数据,并移除 modelscope/ms-swift#10517 / modelscope/ms-swift#10546 中 recipe 层的 planner、手动重排和独立的 balance_batch 开关。异步兼容能力通过公共的引用与元数据调度接口保留,无需单独修复。

验证

以下相关测试共 292 项通过:

  • tests/model/test_dp_balance_backends.py
  • tests/model/test_dp_balance.py
  • tests/model/test_dp_balance_safety.py
  • tests/model/test_micro_batch.py
  • tests/model/test_grpo_micro_batch.py
  • tests/model/test_batch_context_loss.py
  • tests/infra

覆盖梯度不变性与自动跳过回归(含 KTO、Pointwise reranker 及 GRPO / PPO 各控制项)、真实 Ray ObjectRef 调度且控制端不拉取特征、输出顺序还原、单 micro-batch forward_only 后的 calculate_loss、TP / PP / CP 副本分片、与实际 Megatron processor 布局对照的代价估算、嵌套调度,以及两进程 Gloo 下 Megatron 风格 micro-batch 输出的交换与还原。git diff --check 通过;相较 rebase 后的分支基线,无新增 Ruff 诊断。

实验计划

环境:4 × 昇腾 910B(2 卡训练,2 卡 vLLM),Python 3.12,torch 2.10.0 / torch_npu 2.10.0.post4,transformers 5.14.1,vLLM 0.31.0 / vllm-ascend 0.19.1rc2,megatron-core 0.18.2 / MindSpeed 26.2.0,flash-linear-attention 0.5.2,ray 2.59.0。

配置:Qwen3.5-4B GRPO,GSM8K,LoRA(r=16,alpha=32,all-linear),bf16,DP=2。每轮 rollout 8 条 prompt × 8 个采样,每个优化步 32 样本;最大生成长度 2048。FSDP2 使用动态 batching(每个 micro-batch 8192 token);Megatron 使用固定 micro-batch(每 rank 4 行)。同一组训练超参、同一随机种子,仅改变 dp_balance。

后端 dp_balance 训练耗时(ms / 1k tokens) 不均衡度(max / mean) iteration 耗时 平均 reward
FSDP2 False 1335 1.056 464 s 0.448
FSDP2 True 1037(1.29x) 1.011 439 s 0.428
Megatron False 1161 1.108 454 s 0.478
Megatron True 1044(1.11x) 1.027 448 s 0.408

lincq2000 and others added 5 commits November 12, 2026 17:14
forward_backward and forward_only now reorder each slice_dp batch on the
driver (Ray mode) so every DP rank gets a similar workload, reorder every
batch-aligned argument with it, and return the outputs in the caller's
order. TransformersModel(dp_balance=False) turns it off.

- remote_function(dp_balance=True) asks the model's _dp_balance_call for
  a plan; calls with object refs, raw logits, unencoded inputs or lists of
  another length are dispatched unchanged.
- Loss.dp_balance_group_size keeps paired samples together (preference
  and reward losses: 2, REAL: num_generations); embedding and listwise
  reranker losses, and plain callables, opt out.
- calculate_loss rejects per-sample lists after a reordered forward_only.
- dp_balance/* metrics are reported by calculate_metric.
- The GRPO cookbook no longer calls plan_dp_balance itself.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Skip the blocking _dp_balance_spec round trip once a model reports
  dp_balance off; the flag is fixed at construction.
- Do not reorder when the loss for the call cannot be resolved.
- Leave the hidden-size default to estimate_workload.
- Drop the unread wrapper._dp_balance and the one-line micro-batch
  config helper.
- Document dp_balance in the TransformersModel docs (en/zh).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Each rank holds only its own slice, so the ranks all-gather their sequence
lengths, plan the joined batch as the Ray driver would, swap samples when
the plan reorders, and send per-sample outputs back to their rank.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…nment

Skip balancing for rank-local token means, batch-dependent objectives and undeclared losses. Keep audited sample-mean and globally normalized sum configurations enabled.

Reuse aligned loss signals from reordered forward_only calls, reject fresh nested batch arguments and loss-key changes, and fail explicitly when micro-batch caches cannot represent the full batch. Add numerical gradient and cached-call regression coverage and update both language docs.
Move common configuration and loss/backend specification to TrainableModel. Enable inherited dispatch for model entry overrides, preserve replica scheduling, estimate Megatron fixed micro-batch and CP/SP layouts, and resolve only preplanned ObjectRef shards in workers.

Keep unsafe losses, opaque references and unsupported schedules unchanged. Restore local Megatron micro-batch outputs, prevent nested rebalancing, document the shared contract and add Ray/Gloo and backend-layout regression coverage.
@lincq2000
lincq2000 force-pushed the feat/dp-balance-default branch from dd84174 to 4ec848f Compare November 12, 2026 09:35
@lincq2000 lincq2000 changed the title feat(model): balance DP workloads inside TransformersModel by default feat(model): 将 DP 负载均衡内化到 TrainableModel,默认开启并支持 Megatron Nov 12, 2026
GRPO and PPO sequence-mean now keep DP balancing with per-token and
per-sequence controls (dual-clip, overlong filtering, rollout importance
sampling, off-policy sequence masking). Entropy-quantile filtering and
FIPO depend on the local batch and still keep their order.

A reordered forward_only whose rank batch runs as a single in-order
micro-batch now caches the aligned loss arguments, so a following
calculate_loss works as in the unbatched path.

Add gradient-invariance tests for KTO, pointwise reranker and the GRPO
controls.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@lincq2000 lincq2000 changed the title feat(model): 将 DP 负载均衡内化到 TrainableModel,默认开启并支持 Megatron feat(model): move DP balancing into TrainableModel, enabled by default Nov 13, 2026
lincq2000 and others added 2 commits November 14, 2026 09:03
Resolve conflicts in infra/dispatch.py and model/base.py:
- dispatch.py: re-apply dp_balance on top of the _run_here/_call_replicas
  refactor (sticky/exclusive/control/replay/worker_selector were removed upstream).
- base.py: keep both the DP balance helpers and the new response-frame /
  rollout-feature helpers on TrainableModel.

Follow-ups needed for the merged tree to import and test:
- base.py: import get_task_spec from twinkle.processor.tasks
  (twinkle.model.tasks no longer defines it).
- collectors: collect_tensor_dict was renamed to tensor_dict.
- test_dp_balance_safety: give the test harnesses the new
  _rollout_features / resolve_output_frame_request helpers.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Resolve conflicts in infra/dispatch.py and tests/model/test_micro_batch.py:
- dispatch.py: local mode now runs through _run_here; the local DP balance
  hook wraps that entry instead of calling the method body directly.
- test_micro_batch.py: keep the upstream harness collaborators together with
  the attribute-capable optimizer group the cached-signal path needs.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant