Skip to content

[Fix] Correct generic MTP future embeddings and empty-shard loss - #2178

Merged
jayhenry merged 1 commit into
feat/glm53flash-f2-vision-towerfrom
fix/mtp-local-future-embeddings
Oct 15, 2026
Merged

jayhenry merged 1 commit into
feat/glm53flash-f2-vision-towerfrom
fix/mtp-local-future-embeddings

Conversation

@jayhenry

@jayhenry jayhenry commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Stack #2179 (bottom to top):

  1. [Fix] Preserve DeepEP layout events in synchronous dispatch #2177 fix/deepep-layout-event → main
  2. [Feature] Add GLM-5.3-Flash F0: 25B cropped reference checkpoint builder #2105 feat/glm53flash-materialize-full-f0 → fix/deepep-layout-event
  3. [Feature] Add GLM-5.3-Flash F3: Kimi Delta Attention (KDA) #2106 feat/glm53flash-f3-kda → feat/glm53flash-materialize-full-f0
  4. [Feature] Add GLM-5.3-Flash F4: mHC four-stream residual #2107 feat/glm53flash-f4-mhc → feat/glm53flash-f3-kda
  5. [Feature] Add GLM-5.3-Flash F5: NoPE DSA + KPool indexer + clamped SwiGLU #2108 feat/glm53flash-f5-nope-dsa → feat/glm53flash-f4-mhc
  6. [Feature] Add GLM-5.3-Flash F1: VL data preprocessing pipeline #2109 feat/glm53flash-f1-vl-data → feat/glm53flash-f5-nope-dsa
  7. [Feature] Add GLM-5.3-Flash F2: vision tower + projector (eager) #2110 feat/glm53flash-f2-vision-tower → feat/glm53flash-f1-vl-data
  8. [Fix] Correct generic MTP future embeddings and empty-shard loss #2178 fix/mtp-local-future-embeddings → feat/glm53flash-f2-vision-tower ← you are here
  9. [Feature] Add GLM-5.3-Flash F6 core: text model + MTP + compose model #2111 feat/glm53flash-f6-text-moe → fix/mtp-local-future-embeddings

Base is #2110's branch (layer 7). Review only this PR's own diff.


Summary

  • Gather input embeddings once and build only the local packed future slice at each MTP depth, preserving the existing gradient path and packed sequence boundaries.
  • Keep the empty SP shard loss in FP32 while retaining zero gradient edges for MTP outputs.
  • Add public behavior tests for values, gradients, checkpoint replay and empty shards.

This generic MTP layer follows the F2 vision layer. GLM-5.3-Flash specific MTP wiring and integration tests remain in F6.

Tests

  • tests/loss/test_mtp_e2e_tv_loss.py, tests/loss/test_mtp_empty_shards.py and tests/model/test_mtp_local_future_embeddings.py passed in the targeted CPU run.
  • The GLM53 MTP integration case passed in the separate 17-test GPU run.

Co-authored-by: Kuang <128125671+Knlife@users.noreply.github.com>
@jayhenry
jayhenry force-pushed the fix/mtp-local-future-embeddings branch from c56b137 to e850591 Compare October 15, 2026 04:37
@jayhenry
jayhenry merged commit 0afae0c into main Oct 15, 2026
9 of 14 checks passed
@jayhenry
jayhenry deleted the fix/mtp-local-future-embeddings branch October 15, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants