Skip to content

perf(glm): fuse MTP input normalization and concatenation - #1712

Merged
shihaobai merged 1 commit into
mainfrom
bsh/glm-mtp-fused-norm
Nov 13, 2026
Merged

shihaobai merged 1 commit into
mainfrom
bsh/glm-mtp-fused-norm

Conversation

@shihaobai

@shihaobai shihaobai commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Fuse GLM MTP normalization and concatenation before eh_proj in one Triton kernel.

Changes:

  • Fuse the three RMSNorms and concatenation while preserving intermediate rounding and in-place updates.
  • Remove redundant MTP-path branches and checks.
  • Add a single-host GLM-5.3 Flash 2P1D launcher (TP4 for each P/D worker) with BF16 and FP8 KV defaults.

Validation:

  • 39 CUDA tests passed.
  • H200 BF16 microbenchmark, hidden size 4096:
    • 64 rows: 0.0716 ms -> 0.0189 ms (3.78x)
    • 8192 rows: 0.1904 ms -> 0.1147 ms (1.66x)
  • Max absolute difference versus baseline: 0.

Baseline is three RMSNorm launches plus torch.cat; timings exclude eh_proj and are not end-to-end throughput claims.

@shihaobai
shihaobai force-pushed the bsh/glm-mtp-fused-norm branch 2 times, most recently from 62a8a57 to 2860eda Compare November 13, 2026 07:41
@shihaobai
shihaobai force-pushed the bsh/glm-mtp-fused-norm branch from 2860eda to 9caf9b7 Compare November 13, 2026 07:52
@shihaobai
shihaobai merged commit 24ce64d into main Nov 13, 2026
1 check passed
@shihaobai
shihaobai deleted the bsh/glm-mtp-fused-norm branch November 13, 2026 07:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant