Repository navigation
Conversation
Adds xtuner/v1/module/decoder_layer/mhc.py (MHCConfig / hc_split_sinkhorn / hc_pre / hc_post / unshard_hc_params) and xtuner/v1/ops/hc_post.py (fused Triton hc_post_fused), following doc/xtuner_glm5p3flash_design.md F4. The core math is ported from xtuner's dsv4 branch (DeepSeek-V4's Hyper- Connections, commit 01c31a8, not merged into this branch) into the model-agnostic public location the design doc calls for, dropping V4's XTUNER_V4_HF_PARITY global toggle (the default bf16-fast path already degrades to HF-exact math under fp32 inputs, which the tests use as the parity anchor) and the unported 721-line TileKernels backend (left as a documented NotImplementedError gap rather than a blind port with no hardware to validate it against). Adds xtuner/v1/model/moe/glm53/decoder_layer.py: Glm53DenseDecoderLayer (overrides DenseDecoderLayer._forward) and Glm53MoEDecoderLayer, which only overrides MoEDecoderLayer's _pre_moe_forward/_post_moe_forward seams so the ~400-line EP/dispatcher/domino-micro-batch pipeline stays untouched; the mHC residual rides through those methods as an opaque _MHCResidual payload in place of the base class's plain Tensor. mhc_cfg=None degrades both layers to the base class's ordinary residual math unchanged, for reuse by the MTP layer in F6. Also fixes a real bug surfaced while testing: KDA's dt_bias/A_log were left torch.empty-uninitialized in xtuner/v1/module/attention/kda.py; now initialized to match HF's Glm5NextTextForgetGate _init_weights policy (A_log zeroed when a safe gate lower bound is set, dt_bias log-uniform). Verified with 26 tests across tests/model/test_glm53_mhc.py (Sinkhorn math + exact HF parity in fp32, bf16 parity within ULP tolerance), tests/ops/test_hc_post.py (fused kernel vs eager reference, forward/ backward/compile), and tests/model/test_glm53_decoder_layer.py (both decoder layers, mhc_cfg=None passthrough equivalence, gradient flow). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
HF normalizes the flattened mHC streams with the text-config RMS epsilon, not the hyper-connection eps. The hc_pre norm_eps parameter existed but all four call sites relied on the 1e-6 default, so checkpoint gradients drifted from HF. Pass input/post-attention layernorm variance_epsilon at each site; the five-layer crop gradient oracle now agrees with HF within 5%.
…iGLU
Implements the F5 milestone from doc/xtuner_glm5p3flash_design.md: the
NoPE (qk_rope_head_dim=0) DeepSeek Sparse Attention layers, their KPool
indexer (pools of index_kpool consecutive tokens scored together instead
of per-token top-k), a new flash_mla_cudnn SparseMLA backend (FlashMLA
forward + existing cuDNN backward), and the clamped SwiGLU activation
used by GLM-5.3-Flash's dense/shared/MoE MLPs.
- xtuner/v1/ops/act_fn.py: native_clamped_swiglu + MoEActFnConfig support.
- xtuner/v1/module/decoder_layer/{dense,moe}_decoder_layer.py: wire
swiglu_limit through DenseMLP/MoEMLP.
- xtuner/v1/ops/sparse_mla/kpool.py: pool layout, causal ranges, and
top-k pool selection (torch reference + TileLang-backed production
path, reusing the existing indexer kernel unmodified since relu's
homogeneity makes head_dim^-0.5 movable from the relu argument into
the per-head weight without changing the result).
- xtuner/v1/ops/sparse_mla/flash_mla_cudnn.py: new SparseMLA backend.
- xtuner/v1/ops/sparse_mla/tilelang.py: widen the hardcoded 576 head-dim
check to a (head_dim, value_dim) whitelist.
- xtuner/v1/model/moe/glm52/dsa_mla.py: reject flash_mla_cudnn as an
indexer_backend (SparseMLABackend widened, but this backend has no
indexer counterpart).
- xtuner/v1/model/moe/glm53/nope_dsa_mla.py: NoPEDSAMLAConfig,
KPoolIndexer, NoPEDSAMultiLatentAttention.
- xtuner/v1/ops/sparse_mla/protocol.py: KPoolIndexerBackend (restricts
GLM-5.3-Flash's KPool indexer_backend to "torch"/"tilelang" -- unlike
GLM-5.2's 6-way DSAIndexerBackend, there's no cudnn_dsa/flash_mla/
deep_gemm_fp8/cute_dsl KPool kernel) and KPoolTopKIndicesProtocol.
- xtuner/v1/ops/sparse_mla/__init__.py: get_kpool_topk_indices(backend),
mirroring get_dsa_topk_indices's style -- explicitly raises for any
backend other than "torch"/"tilelang" instead of silently falling
through. KPoolIndexer resolves this once in __init__ into
self._topk_indices_fn instead of re-branching on every forward call.
- xtuner/v1/model/moe/glm53/nope_dsa_mla.py: NoPEDSAMultiLatentAttention.
forward branches on freeze_dsa_indexer (torch.no_grad() only when
frozen), mirroring GLM-5.2's per-token indexer, ahead of a future
differentiable indexer output -- today's kpool_topk_indices/
torch_kpool_topk_indices still only ever return an int32 index
tensor, so this doesn't yet change what's trainable (confirmed via a
freeze_dsa_indexer=False smoke run: indexer params get
requires_grad=True but no actual gradient). Calls self.indexer(...)
directly instead of through reuse_during_recompute, which retained
topk_ids' activations across the backward recompute pass to avoid
recomputing them; dropped since nothing currently offloads or reuses
that retained memory.
Known gaps recorded in doc/progress.md: tilelang sparse_mla_backend and
deep_gemm_fp8 indexer_backend are not implemented for NoPE (explicit
NotImplementedError, not silent fallback); KPool/NoPE-DSA sequence-
parallel path is written per design but not yet GPU-tested at sp_size>1;
whether index topk ids can be offloaded is still an open question, not
attempted here.
Test plan: tests/model/test_glm53_dsa.py (12), test_glm53_nope_dsa_mla.py
(2), test_flash_mla_cudnn_sparse_mla.py (3) all pass; GLM-5.2's existing
tests/module/attention/test_dsa_mla.py (15) rerun clean against the
shared tilelang.py/dsa_mla.py edits, confirming no regression.
Numerical oracle: transformers 5.17.0's glm5_next model. Two near-tied
top-k floating-point sensitivities were root-caused via seed sweeps and
a targeted trace script (not worked around by loosening tolerances
blindly) -- see doc/progress.md F5 section for the full analysis.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Also folds in what were originally two follow-up commits, now part of F5 itself
rather than separate history:
- Give the DSA backends explicit defaults instead of inheriting: both
DSAMLAConfig/NoPEDSAMLAConfig fields now default to tilelang directly,
with indexer_backend never falling back to sparse_mla_backend (the two
name different, only partially overlapping backend vocabularies).
- Bound the KPool indexer's logits tile with query chunking, mirroring the
existing tilelang DSA selector's query-chunk bound.
Implements the F1 milestone from doc/xtuner_glm5p3flash_design.md: the
chat template, TokenizeFn (cache/runtime dual path), DataItem, and
Collator/SequenceContext plumbing for GLM-5.3-Flash's image/video SFT
data pipeline. Authority for the F1 contract is duanyanhui's
xtuner_glm53_flash_vl_vision_design.md §7-8 and
xtuner_glm53_flash_vl_develop_stages.md §3 (stage 2).
- xtuner/v1/data_proto/messages/glm53_chat.py: render_glm53_chat /
Glm53ChatMessages, adapted from the already-landed glm52_chat.py at
exactly its two documented diffs (reasoning_effort domain
{high,max}->{low,high,max}; media renders as the two-stage
<|begin_of_image|><|image|><|end_of_image|>-style placeholder marker
instead of GLM-5.2's <reminder> text). Verified byte-for-byte against
the real chat_template.jinja via tokenizer.apply_chat_template.
- xtuner/v1/datasets/mllm_tokenize_fn/glm53_vl_tokenize_fn.py:
Glm53VLTokenizeFunction/Config. Placeholder expansion and
mm_token_type_ids reuse the real Glm5NextProcessor's own public
replace_image_token/replace_video_token/create_mm_token_type_ids in
both cache (geometry-only, no media decode) and runtime paths, so the
two paths run the identical HF code rather than two independent
reimplementations. Mixed image+video in one sample raises
NotImplementedError instead of silently dropping video.
- xtuner/v1/datasets/data_item.py: Glm53VLDataItem.
- xtuner/v1/data_proto/sequence_context.py: pixel_values_videos /
video_grid_thw / mm_token_type_ids fields, wired through
__init__/split/cat/to/copy/data.
- xtuner/v1/datasets/collator.py: build_text_ctx_labels now also
returns kept_indices (which original instances survived pack-level
truncation); glm53_vl_sft_collator built on top.
Root-caused two real discrepancies via direct package inspection /
minimal repro rather than trusting the design doc's claims: (1)
transformers 5.17.0 has no Glm5NextVideoProcessor.get_number_of_video_
patches despite the design doc and an independent peer check both
asserting it exists -- verified missing via hasattr/grep across the
installed package, and confirmed the intended cache-path geometry
prediction (Glm5NextVideoProcessor.sample_frames + the module-level
smart_resize -- the same two public calls the real preprocessing path
makes) reproduces the real video_grid_thw bit-for-bit; (2) an earlier
version expanded the whole <|begin_of_image|>...<|end_of_image|> marker
instead of just the inner <|image|> token, undercounting num_tokens by
2 per media item -- caught by an end-to-end diff against a real
processor() call, fixed by only replacing the bare inner token.
doc/xtuner_glm5p3flash_design.md / .py: synced the F1.c reference
table and the cache-path docstring to reflect finding (1) above --
these still claimed both get_number_of_image_patches and
get_number_of_video_patches were exposed pure-geometry APIs, which was
true only for the image side; the video side needs the sample_frames +
smart_resize fallback described above. This was an oversight in the
original commit: the code and this commit message already reflected
the real finding, but the design doc's own reference tables didn't get
updated to match at the time.
Known gap: DataloaderConfig.build_collator() doesn't yet register
glm53_vl_sft_collator (needs image_token_id/merge_unit threaded from
the tokenize_fn instance) -- left for F6's end-to-end wiring.
Test plan: tests/datasets/test_glm53_chat.py (6, byte-exact vs HF
jinja), test_glm53_vl_collator.py (9, includes build_text_ctx_labels
regression coverage for qwen3_vl_sft_collator/intern_s1_vl_sft_collator
since they share the builder), test_glm53_vl_tokenize_fn.py (10,
includes two tests asserting input_ids/mm_token_type_ids match a real
end-to-end processor() call token-for-token). All 25 pass, CPU only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`glm53_vl_sft_collator` landed with F1 but was never reachable from a training config: `DataloaderConfig.build_collator()` only knew the three older collators by name, and the dotted-path fallback could not supply the two extra arguments this collator needs (`image_token_id`/`merge_unit`, used to re-check the placeholder<->patch-count invariant after pack-level truncation). Any GLM-5.3-Flash VL config therefore died in the dataloader with a `TypeError` before the first step -- the F1 "known gap" that blocked end-to-end VL training. Registers the name, and adds a generic `collator_kwargs` that is bound into the collator partial for any collator needing more than the four arguments the dataloader supplies itself. The name->function dispatch collapses into a lookup table so every branch, including the dotted-path one, goes through the same binding. Test Plan: `test_glm53_collator_is_reachable_by_name_from_dataloader_config` builds the collator through `DataloaderConfig` and calls it with only the dataloader-supplied arguments; the other four names plus a dotted path still resolve. tests/datasets/test_glm53_vl_collator.py 7/7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`xtuner.v1.datasets.mllm_tokenize_fn` re-exported it but the package root did not, unlike every other VL tokenize-fn config, so a training config had to reach into the submodule by path. Makes the GLM-5.3-Flash VL entry point look like Qwen3-VL's and InternS1's. Test Plan: `from xtuner.v1.datasets import Glm53VLTokenizeFnConfig, Glm53VLTokenizeFunction`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The class documented that GLM-5.3-Flash renders through `render_glm53_chat` instead of a `CHAT_TEMPLATE_MAP` entry, and `build()` never reads `chat_template` -- but the field is declared required on `BaseMLLMTokenizeFnConfig`, so constructing the config the documented way raised `ValidationError: chat_template Field required` before any data was touched. Defaults it to the template name the renderer implements, which keeps the field's declared type while making the documented construction work. Test Plan: `Glm53VLTokenizeFnConfig(processor_path=..., max_length=...)` constructs; exercised end to end by the VL SFT config in the F6 layer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The VL e2e matrix showed `tgs` at roughly half of `seqlen_tgs` on every profile, where the text matrix has them within 0.05%. The two differ only by padding (`mask.sum()` vs `mask.numel()`), so VL packs were ~50% padding -- one step packed 131 real tokens into a 16384 slot. The dataset was large enough to fill them (2.38M tokens against 2.62M slots), so this was not starvation. Comparing each sample's cached `num_tokens` (which the packer budgets with) against its runtime length found 17% of samples cached at 4096 but training as **33-token fakes**. Two defects, both in the cache/runtime split, compounded: 1. **Frame-folder videos were indexed against the original video.** Both paths built `VideoMetadata` from `origin_video_length`/`origin_fps` (182 frames @30fps), but a frame folder holds the video *after* resampling -- `processed_video_length` frames at `processed_fps` (23 @4fps). Runtime indexed a 23-file folder with original-video frame numbers and raised `IndexError: list index out of range`; the dataset swallowed it and substituted a fake sample. Sampling now uses the processed sequence for frame folders, qwen3_vl's `calc_frame_info` rule 3, which also makes the frame timestamps correct. The folder/file decision is read from the path string (the trailing `/` convention qwen3_vl's reader keys on), not the filesystem, so the cache path -- which never touches the media -- reaches the same answer as the runtime path. 2. **Over-length visual samples were packed at their truncated length.** With the index fixed, the same samples expand to 14,352 placeholders, far past `max_length=4096`. Truncating text is harmless, but truncating *through* a visual span leaves fewer placeholders than patches; the cache path reported the truncated 4096 anyway, so the packer budgeted for it and runtime then rejected it -- again becoming a fake. The cache path now reports such samples as damaged (`num_tokens=0`) so they are filtered before packing, the same check qwen3_vl's cache path makes ("如果图片被截断，则该数据丢弃"). Both escaped the existing tests for concrete reasons: `_video_item` only set `origin_*` on a folder holding exactly that many frames, so origin and processed coincided; and the truncation test only asserted the *runtime* path raises -- which is exactly the behavior that gets silently turned into a fake sample, while the cache path, where the damage is decided, was never checked. Test Plan: - `test_glm53_frame_folder_samples_the_processed_frames`: a folder resampled from a longer original (180 frames @30fps -> 6 @2fps); cache and runtime agree and runtime does not raise. - `test_glm53_visual_truncation_is_dropped` now also asserts the cache path returns `num_tokens == 0`, keeping the runtime rejection as a last line of defence. - Both fail on the previous code with the exact production errors (`IndexError: list index out of range`; `assert 20 == 0`) and pass with the fix. tests/datasets/test_glm53_vl_tokenize_fn.py 11/11. - Over the whole ci_vl corpus: 0 cached/runtime length mismatches across the 112 kept samples, with 24 over-length samples filtered before packing (previously 10 of the first 60 mismatched). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implements the eager-precision core of the F2 milestone from doc/xtuner_glm5p3flash_design.md: the native XTuner vision tower and projector for GLM-5.3-Flash, split per the design doc's module boundary (xtuner/v1/model/compose/glm53/). Authority for the F2 contract is duanyanhui's xtuner_glm53_flash_vl_vision_design.md §4-§6/§9-§13 and xtuner_glm53_flash_vl_develop_stages.md §2/§4/§5/§6 (stages 1/3/4/5); this delivery covers stages 1 and 3 (skeleton + weight mapping, eager bitwise forward) in full and stages 4/5 (FSDP2/compile, Vision SP) as wiring only, not production-grade verification -- see Known gaps below and in doc/progress.md. - glm53_config.py: Glm53VisionConfig/Glm53ProjectorConfig, fields checked against the real checkpoint's vision_config (num_heads not num_attention_heads, out_hidden_size not text_hidden_size, rope_parameters defaulted since the checkpoint JSON omits it). - modeling_vision.py: Glm53VisionModel (patch_embed -> 2D axial RoPE -> 24 blocks with RMSNorm+QK-norm+attention+clamped-SwiGLU MLP -> post_layernorm). cu_seqlens/position_ids are natively reimplemented rather than imported from transformers.vision_utils at runtime, per the "native implementation, not an HF training-time dependency" principle (§3.2) and matching qwen3_vl's own established pattern. - modeling_projector.py: Glm53Projector (downsample Conv2d + merger). XTuner's tower/projector split moves downsample into the projector, unlike HF's single Glm5NextVisionModel -- confirmed intentional via the design doc's HF-key-mapping table (§6.1), not an HF module boundary. - vision_utils.py: flatten_video_grid_thw. Test plan: tests/model/test_glm53_vision.py (8 tests, CPU only). Real- checkpoint weight mapping (gated, skips without GLM_5_3_FLASH_PATH): vision_tower (339) + projector (8) = 347 parameters, matching the design doc's independently-recorded real-checkpoint vision key count; from_hf(strict=False) leaves no missing/unloaded keys, spot-checked tensors are torch.equal to the raw safetensors. Forward parity vs HF Glm5NextVisionModel (tiny synthetic config, eager attention, rtol=0 atol=0): single image, multi-tubelet video (grid=[2,4,4], exercises multi-segment cu_seqlens and t-repeated position ids), multi-image batch, plus a backward gradient-flow smoke check. Ground truth is HF model.visual as a whole (§5.2): XTuner's post_layernorm output is compared against HF's pre-downsample state (recomputed via HF's own downsample module, not just pooler_output). Known gaps (design doc's own stage split -- stages 4/5 need multi-GPU coordination -- recorded explicitly, not silently skipped): FSDP2 wiring (fully_shard) mirrors qwen3_vl's structural pattern but has no multi-GPU parity test; attention supports sequence_parallel_mesh with Ulysses all-to-all on q/k/v but the full merge-aligned-padding + local-projector Vision SP design (§9.3/§9.6) and its SP=2/4 parity suite are not implemented; torch.compile config not wired; production attention kernel (FlashAttention/FlexAttention) tolerance matrix (§11.7) not verified, only eager_attention; HF save round-trip not verified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bf16 parity needs the HF/Automodel cast-then-affine norm. Ulysses without merge-aligned padding computes the wrong attention, so size>1 now errors. Co-authored-by: Cursor <cursoragent@cursor.com>
…h kernel path
Closes three F2 known gaps that together kept the vision tower off the production training
path.
**Vision sequence parallel.** The tower previously rejected `sequence_parallel_mesh.size()>1`
outright: Ulysses on an unsharded patch sequence computes the wrong attention, and nothing split
the patches. Implements the design doc's §9.3/§9.6 merge-aligned padding + local projector:
`_shard_patches_for_sequence_parallel` pads the global patch sequence up to a multiple of
`sp_size * spatial_merge_size ** 2` and splits it contiguously, so (a) every shard holds a whole
number of 2x2 merge blocks and can therefore run the projector on its own shard, and (b) all
ranks carry the same patch count, which Ulysses all-to-all requires. The padding is declared as
an extra `grid_thw` row instead of being appended silently, which keeps it a separate
`cu_seqlens` segment -- padding patches can never attend to, or be attended by, a real image.
Position ids are sharded to match the local patches (RoPE is applied before the all-to-all)
while `cu_seqlens` stays global (the all-to-all hands attention the whole sequence).
**Flash attention was broken, not just unverified.** `attn_impl="flash_attention"` died with
`Expected a value of type 'Tensor' for argument 'max_seqlen_q' but instead found type 'int'`:
the tower was the only attention call site materializing `max_seqlen` via `int(...item())`,
which `flash_attn_varlen_func` v2 rejects outright and which also forces a device sync. Now
passed as a tensor like every other call site does with `SequenceContext.max_length_q`.
**Compile config.** `default_compile_cfg` was empty for both modules, so the vision half stayed
eager even under `MODEL_COMPILE=1`. The ViT block takes `fullgraph=False` (it dispatches into an
attention kernel, and under SP it also contains collectives); the projector, which is a Conv2d
plus the merger's clamped-SwiGLU chain with no dispatch or collective, takes a full graph.
Test Plan:
- `TestGlm53VisionSequenceParallel::test_sp_tower_and_projector_match_non_sp` (2 GPU): 20
patches over two images -- deliberately not a multiple of `sp_size * merge_unit`, so the
padding branch is exercised -- tower + projector under SP, gathered and trimmed, matches the
non-SP result.
- `test_unaligned_patch_count_is_rejected_under_sp`: a patch count that would split a merge
block raises instead of silently mis-merging.
- `TestGlm53VisionProductionAttention::test_flash_attention_matches_eager` (GPU): the flash
path now runs and agrees with the HF-aligned eager baseline in bf16.
- tests/model/test_glm53_vision.py 13/13.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…edding `SequenceContext` deliberately does not move `pixel_values`/`pixel_values_videos` in its `.to()` (documented there as "sharded by the model side, then moved"), so the tower receives CPU tensors in real training and died on the very first step with `Input type (CPUBFloat16Type) and weight type (CUDABFloat16Type) should be the same`. Every unit test passed device tensors in directly, so nothing caught it. The move happens after the SP split, not before, so only this rank's shard is transferred -- the reason `SequenceContext` leaves the tensor alone in the first place. It follows the module's own parameter device rather than the global one, so a CPU-resident tower (used by the parity tests) keeps working. Test Plan: `test_cpu_pixel_values_are_moved_to_device` feeds CPU `pixel_values` with a device-resident `grid_thw` -- the exact shape `SequenceContext` produces -- to a CUDA tower. tests/model/test_glm53_vision.py 14/14. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The vision tower never applied activation checkpointing, so all 24 ViT blocks retained their activations. That is not a small oversight for this model: the tower runs over *raw* patches, and a packed VL sample carries far more of them than the LLM sequence has tokens -- an 8192-token pack measured `img_tokens: 14688` -- so the retained attention plus 4096-wide MLP activations of 24 blocks dominate the step's peak memory. End-to-end VL SFT on the F0 25B checkpoint OOMed in backward at `PACK_MAX_LENGTH=16384`, and still OOMed on step 2 at 8192, before the language tower was anywhere near its limit. Wires the blocks through `apply_activation_checkpointing` honouring `vision_recompute_ratio` (default 1.0) and `checkpoint_preserve_rng_state`, the same knobs qwen3_vl's tower already uses. Test Plan: tests/model/test_glm53_vision.py 14/14 (the checkpointed path keeps HF bitwise parity and the SP parity result); end-to-end VL SFT memory result recorded in doc/progress.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wuyh11
force-pushed
the
fix/glm53-fsdp-sp-gradients
branch
from
October 9, 2026 11:32
11b18eb to
747b5b5
Compare
Wuyh11
changed the title
Base this fix on the F2 vision-tower branch so the later text/compose layer can inherit the shared training corrections. Expand FP32 parameters replicated on an EP/TP submesh to the full world mesh before excluding them from FSDP. Keep full-world replicas and sharded DTensors unchanged. Use an autograd-aware gather for SequenceContext.raw_inputs_embeds so future-token inputs retain gradients across SP shard boundaries. Keep the existing real CPU vision RoPE initialization and make its FP32 dtype and non-persistent-buffer rationale explicit.
Wuyh11
force-pushed
the
fix/glm53-fsdp-sp-gradients
branch
from
October 10, 2026 12:35
747b5b5 to
b715eb9
Compare
Wuyh11
changed the title
Wuyh11
changed the base branch from
feat/glm53flash-f6-text-moe
to
feat/glm53flash-f2-vision-tower
October 10, 2026 12:35
jayhenry
force-pushed
the
feat/glm53flash-f2-vision-tower
branch
from
October 10, 2026 19:01
72ce9ed to
99266d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fix FP32 replica synchronization and SP embedding gradients for GLM-5.3-Flash F2.1, targeting
feat/glm53flash-f2-vision-tower(#2110).Root Cause
SequenceContext.raw_inputs_embedsused ordinaryall_gatherunder SP, detaching the embedding-input branch used by future-token consumers such as MTP.Fix
base.py: Expand FP32 replicas from an EP/TP submesh to the full world mesh before adding them to FSDP's ignored parameters. Detach the local DTensor value when creating the replacement leaf parameter; retain existing full-world replicas and sharded placements.sequence_context.py: Useall_gather_tensor_autogradfor embeddings so backward returns gradients to their owning SP ranks through reduce-scatter SUM, including contributions crossing shard boundaries. Cache the gathered tensor with its graph attached.modeling_vision.py: Spell the RoPE frequency dtype astorch.float32and document why the non-persistent buffer must remain real during meta construction.Tests
git diff --checkpass for the three-file diff.2.5. FSDP wrapping is stubbed in these CPU checks.pre_commitis not installed.Verification (8xH200, prior validation)
September 28-30 validation of the shared-utility fixes on an earlier GLM53 training snapshot:
[1, 2, 3, 4]reduced to1.5/3.5before the fix and2.5on every rank afterward, with matching norms and clipping results.The gradient comparison covers all 347 vision/projector parameters before clipping. Strict numerical equivalence across SP sizes remains unresolved. Validation focuses on missing gradient contributions and replica consistency; no before/after throughput benchmark was run.