[Feature] Add GLM-5.3-Flash F2: vision tower + projector (eager)

[jayhenry]

Stack #2179 (bottom to top):

1. [Fix] Preserve DeepEP layout events in synchronous dispatch #2177 fix/deepep-layout-event → main
2. [Feature] Add GLM-5.3-Flash F0: 25B cropped reference checkpoint builder #2105 feat/glm53flash-materialize-full-f0 → fix/deepep-layout-event
3. [Feature] Add GLM-5.3-Flash F3: Kimi Delta Attention (KDA) #2106 feat/glm53flash-f3-kda → feat/glm53flash-materialize-full-f0
4. [Feature] Add GLM-5.3-Flash F4: mHC four-stream residual #2107 feat/glm53flash-f4-mhc → feat/glm53flash-f3-kda
5. [Feature] Add GLM-5.3-Flash F5: NoPE DSA + KPool indexer + clamped SwiGLU #2108 feat/glm53flash-f5-nope-dsa → feat/glm53flash-f4-mhc
6. [Feature] Add GLM-5.3-Flash F1: VL data preprocessing pipeline #2109 feat/glm53flash-f1-vl-data → feat/glm53flash-f5-nope-dsa
7. [Feature] Add GLM-5.3-Flash F2: vision tower + projector (eager) #2110 feat/glm53flash-f2-vision-tower → feat/glm53flash-f1-vl-data ← you are here
8. [Fix] Correct generic MTP future embeddings and empty-shard loss #2178 fix/mtp-local-future-embeddings → feat/glm53flash-f2-vision-tower
9. [Feature] Add GLM-5.3-Flash F6 core: text model + MTP + compose model #2111 feat/glm53flash-f6-text-moe → fix/mtp-local-future-embeddings

Base is #2109's branch (layer 6). Review only this PR's own diff.

---

Summary

Stack layer 7/9 of GLM-5.3-Flash support (base: layer 6, F1 VL data pipeline). The F2 vision tower and projector support production training.

The vision checkpoint test environment update previously in F6 is now included in this layer.

F2 core

Native XTuner vision tower and projector (xtuner/v1/model/compose/glm53/), split per the design doc's §5.2 module boundary: HF's single Glm5NextVisionModel becomes vision_tower (patch_embed → 2D axial RoPE → 24 blocks → post_layernorm) plus multi_modal_projector (Conv2d downsample + clamped-SwiGLU merger). cu_seqlens/position_ids are reimplemented natively rather than imported from transformers.vision_utils, matching qwen3_vl's established pattern. The vision RMSNorm follows HF/Automodel's cast-then-affine rounding, which matters in bf16.

Vision sequence parallel (design doc §9.3/§9.6)

Previously the tower rejected sequence_parallel_mesh.size() > 1 outright — the attention had half an Ulysses path but nothing sharded the patch sequence, which silently computes the wrong attention. Now implemented properly:

• _shard_patches_for_sequence_parallel pads the global patch sequence up to a multiple of sp_size * spatial_merge_size ** 2 and splits it contiguously. Merge alignment is a hard constraint: the projector's downsample treats each consecutive group of 4 patches as one 2×2 block, so a shard boundary inside a group would merge patches owned by two different ranks — alignment is also what lets each rank run the projector on its own shard (local projector). Equal length is what Ulysses all-to-all requires.
• The padding is declared as an extra grid_thw row rather than appended silently, so it forms its own cu_seqlens segment: padding patches can never attend to, or be attended by, a real image.
• Position ids are sharded to match the local patches (RoPE runs before the all-to-all); cu_seqlens stays global (after the all-to-all, attention sees the whole sequence).

Three production bugs

1. attn_impl="flash_attention" was broken, not merely unverified. It died with Expected a value of type 'Tensor' for argument 'max_seqlen_q' but instead found type 'int' — the tower was the only attention call site materializing max_seqlen via int(...item()), which flash_attn_varlen_func v2 rejects and which also forces a device sync. Now passed as a tensor, like every other call site does with SequenceContext.max_length_q.
2. pixel_values was never moved to the device. SequenceContext deliberately leaves the media tensors on CPU (documented there as "sharded by the model side, then moved"), so real training died on step 1 with Input type (CPUBFloat16Type) and weight type (CUDABFloat16Type) should be the same. Every unit test passed device tensors in, so nothing caught it. The move happens after the SP split, so only this rank's shard is transferred.
3. The ViT blocks had no activation checkpointing — the actual cause of multimodal OOM. The tower runs over raw patches, and a packed VL sample carries far more of them than the LLM sequence has tokens (an 8192-token pack measured img_tokens: 14688; a 16384-token pack ~31k), so 24 blocks of retained attention + 4096-wide MLP activations dominated peak memory. VL SFT OOMed in backward at 16k pack and still died on step 2 at 8k, long before the language tower was near its limit. Now wired through apply_activation_checkpointing honouring vision_recompute_ratio/checkpoint_preserve_rng_state, the same knobs qwen3_vl's tower uses. Peak drops to 85.85 GB at 16k pack.

Compile config

default_compile_cfg was empty for both modules, so the vision half stayed eager even under MODEL_COMPILE=1. The ViT block takes fullgraph=False (it dispatches into an attention kernel, and under SP it contains collectives); the projector takes a full graph.

Test Plan

tests/model/test_glm53_vision.py 14/14:

• Real-checkpoint weight mapping: 339 + 8 = 347 parameters, from_hf(strict=False) with no missing/unloaded keys.
• Forward parity vs HF Glm5NextVisionModel at rtol=0, atol=0 (fp32 and bf16): single image, multi-tubelet video, multi-image batch, plus backward gradient flow.
• TestGlm53VisionSequenceParallel::test_sp_tower_and_projector_match_non_sp (2 GPU): 20 patches over two images — deliberately not a multiple of sp_size * merge_unit, exercising the padding branch — tower + projector under SP, gathered and trimmed, matches the non-SP result.
• test_unaligned_patch_count_is_rejected_under_sp: a patch count that would split a merge block raises instead of mis-merging.
• test_flash_attention_matches_eager (GPU): the flash path runs and agrees with the HF-aligned eager baseline.
• test_cpu_pixel_values_are_moved_to_device: CPU pixel_values with a device grid_thw, the exact shape SequenceContext produces.

Known gaps

Vision-side FSDP2 multi-GPU parity is still covered only by the end-to-end smoke, not a dedicated test_vision_fsdp_parity.

[jayhenry]

复用已有的 BaseModel.build_rotary_embedding 是否更好？