Skip to content

fix(kernel): align fused entropy backward with native rounding - #389

Merged
kevssim merged 1 commit into
modelscope:mainfrom
diaochengxi:codex/fix-fused-entropy-normalization
Nov 18, 2026
Merged

kevssim merged 1 commit into
modelscope:mainfrom
diaochengxi:codex/fix-fused-entropy-normalization

Conversation

@diaochengxi

@diaochengxi diaochengxi commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why this PR is needed

PR #385 added the Ascend fused lm_head + selective log-softmax path and optional entropy output for task='fused_lm_rl'. That feature is useful, but its entropy backward does not satisfy Twinkle's strict BF16 gradient-parity requirement on all supported shapes.

This is not a test-only issue and it is not a general failure of the forward path. It is a silent numerical defect in the gradient sent back to the transformer backbone:

  • log-probabilities and entropy outputs pass the BF16 reference tolerance;
  • grad_weight passes;
  • grad_x can contain a small number of out-of-tolerance elements;
  • the same run can otherwise complete normally, without NaN, Inf, or an exception;
  • therefore a normal training run can look healthy while its backbone gradients differ from native PyTorch in exactly the path that controls RL fine-tuning.

This PR is a focused follow-up because #385 is already merged into main. It fixes the defect without changing the public API, adding model parameters, or changing the logp-only fast path.

Trigger conditions

The defect is reached only when all of the following are true:

  • task='fused_lm_rl' is used;
  • the RL loss requests entropy, currently entropy_coef > 0 in GRPOLoss/subclasses;
  • a backward pass is executed and grad_x is needed by the backbone;
  • NPU + BF16 + Triton selects the fused kernel;
  • the vocabulary and logits are large enough for the approximation error to survive final BF16 rounding.

FP32, non-NPU, Triton-unavailable, non-fused, logp-only, and inference-only paths are unaffected.

Observed failure on current main

Reproduction configuration:

device: Ascend A2, 32 GB HBM
rows: 256
hidden: 2048
vocab: 151936
dtype: BF16
temperature: 1.7
seed: 20261117

Acceptance threshold:

torch.testing.assert_close(rtol=1.6e-2, atol=1e-2)

On current main, the added regression test fails in the backward comparison:

Mismatched elements: 2 / 524288 (0.0%)
Greatest absolute difference: 0.012451171875
Greatest relative difference: 3.671875

The exact mismatch count depends on the random output-gradient seed. Across the validation runs it ranged from 2 to 28 out of 524288, with a maximum absolute difference of 0.03125 in one configuration. Other outputs in the same comparison pass, which makes the failure easy to miss unless the joint gradient check is run.

Root cause

The fused entropy backward approximated probabilities as:

probabilities = exp(logits - lse)

However, the row-wise value used by the forward reduction and the value used by backward were not normalized from the same max + denominator representation. On large logits this creates a small bias in E[logits], which is part of the entropy gradient:

grad_logits = p * (-grad_logp + grad_entropy * (E[logits] - logits))

That small bias is then amplified by the final input-gradient GEMM. Near cancellation points, the result lands one or more BF16 ULPs away from native PyTorch and exceeds the fixed atol.

There was a second, independent issue: native PyTorch rounds the entropy gradient to BF16 at the scaled-logits/temperature boundary before running the linear backward. The fused path previously did not reproduce that rounding order exactly.

Fix

  • Save row_max and row_sum from the fused forward reduction and reuse them for normalized probabilities:
    exp(logits - row_max) / row_sum.
  • Use FP32 softmax for entropy backward so E[logits] follows the native reference path.
  • Reproduce native BF16 rounding order: compute the entropy gradient in FP32, cast at the scaled-logits boundary, apply temperature division, then run the gradient GEMM.
  • Keep the existing Triton backward path for logp-only training; the correctness fix is scoped to the entropy branch.
  • Add a seeded large-vocabulary regression test that fails on current main and passes with the fix.

Verification

The fixed branch passes the strict BF16 comparison:

Implementation grad_x mismatches Max absolute difference
Current main 2-28 / 524288 up to 0.03125
Fixed branch 0 / 524288 0.00390625

The same strict check also passes at rows=1024.

Test suites run on the A2 validation host:

tests/kernel/ops/test_fused_linear_selective_log_softmax.py
tests/transformers/test_fused_rl_logps.py

49 passed

Performance impact

Fused entropy, rows=256, hidden=2048, vocab=151936:

Implementation Mean latency Peak allocated memory
Current main 40.88 ms 3.296 GB
Fixed branch 28.66 ms 3.687 GB
Native PyTorch reference 17.25 ms 5.455 GB

The fix is approximately 1.43x faster than the currently merged fused implementation and uses about 32% less peak memory than native PyTorch. It is still slower than native PyTorch in this entropy configuration; this PR does not claim a native-level single-op speedup.

End-to-end Qwen3-0.6B, B=2, S=512, 50 measured steps, entropy_coef=0.01:

Mode Mean step Peak memory Max loss difference vs native
Native 200.99 ms 7.25 GB 0
Current fused entropy 299.35 ms 4.48 GB 0.00396
Fixed fused entropy 257.62 ms 6.04 GB 0.00478

The fixed branch improves step time over the broken fused path while remaining within a 0.005 loss difference from native over the measured run.

Merge / maintenance assessment

  • No public API or model-parameter changes.
  • No new configuration surface.
  • The logp-only fused path remains unchanged.
  • The entropy branch intentionally prefers native numerical semantics over a slightly lower-level Triton approximation; this is required by the acceptance contract.
  • The new test is NPU-only and skipped automatically when torch_npu is unavailable.

@diaochengxi
diaochengxi force-pushed the codex/fix-fused-entropy-normalization branch from 4e0007a to bdc34f8 Compare November 17, 2026 09:12
@kevssim
kevssim merged commit e8b858d into modelscope:main Nov 18, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants