[Fix] Align GLM53 multimodal inputs with the native HF protocol

[ShilohYu]

GLM53 training can diverge from the native HF input protocol when enable_thinking=False, when a video frame directory uses names such as frame_2.jpg and frame_10.jpg, or when truncation preserves all visual placeholders but removes a media end marker. This fixes those cases and compares full inputs against the HF template independently of XTuner's renderer.

Base: InternLM/xtuner:feat/glm53flash-f1-vl-data at a29e63978 (#2109; rebased onto the force-pushed F1 branch on 2026-10-15, previously e933fdcf5). This is a focused follow-up to F1. After merge, the normal stack integration should propagate it through F2 (#2110), F6 (#2111), and the preview branch. It does not rewrite those branches.

Root Cause

• The GLM53 renderer inherited GLM52's enable_thinking behavior, while the published GLM53 template always emits the reasoning-effort header and an open generation-time <think> prefix.
• GLM video directory loading used lexicographic sorting, which mismatches numeric frame order while retaining chronological timestamps.
• Placeholder counts alone cannot detect a missing image/video closing marker. The collator can also remove that closing marker through its final next-token shift or truncate an oversized multimodal sample after tokenize-time validation.

Fix

• Match the published GLM53 template for both values of enable_thinking, keeping the parameter for API compatibility and supervising the closing thought marker.

• Reuse the existing numeric frame sorter. Frame directories must use <number>.jpg or <prefix>_<number>.jpg; invalid names fail explicitly.

• Check image and video boundary completeness on the model-input prefix in both cache and runtime paths. Cache filtering rejects damaged spans before packing; runtime checks reject bypasses.

• Reject an oversized multimodal sample in the GLM collator, requiring truncation to happen in the GLM tokenizer where media boundaries can be validated. Other collators are unchanged.

• Add independent native-template comparisons for text, image, multiple images, multi-turn SFT, generation prefixes, numbered frame folders and MP4 inputs, including pixels, grid, token IDs and modality IDs.

• Note on length caches: the cache key is intentionally unchanged. The corrected rendering/filtering can change per-sample lengths and filtering decisions, so if you ran training with pre-fix code, delete the existing dataset length cache once after upgrading. Runs started after this fix are unaffected.

• Use any(num_img_tokens) for the collator media predicate, fixing the reported mypy error without changing its nonnegative-count semantics.

Official HF source evidence

All links below pin immutable revisions. The audited local chat template is byte-for-byte identical to zai-org/GLM-5.3-Flash@3f1971b7b5f7a528c9c4ef6212c8785298a8c24a (SHA-256 34d5ee66b12fa6446cdae131c352b8f68cd85369e0e6fda115583805fada3891). Transformers v5.17.0 resolves to commit 856157a2f3e9594954310df18fdccc31ffddebe9; the four cited Python files were downloaded from that commit and verified byte-for-byte against the installed test environment.

1. Reasoning prefix follows the published template

The official reasoning-effort header defaults to max and always emits the system header; it does not inspect enable_thinking:

{%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%}
{%- if effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}

The official generation prefix is always an open <think>:

{%- if add_generation_prompt -%}
    <|assistant|>{{- '<think>' -}}
{%- endif -%}

For an existing assistant answer, lines 149–153 render <think></think> when no reasoning is present. This directly supports the rendered-text fix. Supervising </think> is XTuner's SFT loss-mask decision so that the model can learn to close the open HF inference prefix; the HF Jinja template itself does not define training labels.

2. Video pixels and timestamps must use the same frame indices

HF Decord decoding uses the selected indices for both the pixels and metadata.frames_indices:

    indices = sample_indices_fn(metadata=metadata, **kwargs)
    video = vr.get_batch(indices).asnumpy()
# The same indices are stored in metadata.frames_indices at line 453.

VideoMetadata.timestamps derives each timestamp from those indices:

return [frame_idx / self.fps for frame_idx in self.frames_indices]

Glm5NextVideoProcessor.sample_frames selects chronological frame indices. Glm5NextProcessor.replace_video_token then uses metadata.timestamps[::2] and emits frame text in that order.

HF does not prescribe our frame-directory filename convention. Numeric sorting is the XTuner adapter needed to preserve this pixel/index/timestamp correspondence when frames are stored as frame_<N>.jpg. A lexicographic listing breaks it (frame_10 precedes frame_2). The numbered-folder regression compares pixels as well as token IDs; timestamp text alone cannot detect that bug.

3. Media wrappers are part of the HF input protocol

The official template media macros create complete wrappers:

{%- macro emit_image() -%}<|begin_of_image|><|image|><|end_of_image|>{%- endmacro -%}
{%- macro emit_video() -%}<|begin_of_video|><|video|><|end_of_video|>{%- endmacro -%}

replace_image_token expands the inner image placeholder to prod(image_grid_thw) // merge_size**2 tokens. replace_frame_token_id emits each video's image wrapper and timestamp. The outer begin/end markers remain part of the sequence.

The video boundaries also affect create_mm_token_type_ids:

            starts = np.cumsum(array_ids == self.video_start_id, axis=0)
            ends = np.cumsum(array_ids == self.video_end_id, axis=0)
            is_video_modality = starts > ends

            mm_token_types[(array_ids == self.image_token_id) & is_video_modality] = 2
            mm_token_types[(array_ids == self.image_token_id) & (~is_video_modality)] = 1

HF's generic _check_special_mm_tokens rejects changes in placeholder counts after tokenization, warning about truncation. It does not implement XTuner's complete-wrapper/next-token-shift check. The new boundary guard and the oversized-sample rejection are XTuner-specific protections that preserve the complete native HF sequence across cache, truncation, packing and label shift; they are not claimed to copy an identical HF truncation algorithm.

Template revision scope

The currently published BF16 template at a5b45eb4 differs from the audited snapshot in null-content handling and tool-call rendering/loop control. The reasoning header, generation prefix and media macros cited above are unchanged. This PR does not claim full parity for those newer tool/null cases.

AutoModel cross-check

CPU comparison against AutoModel eb11d83ece9952b12e2f770df9fc524c1cbfe6fb using the same checkpoint assets and Transformers 5.17.0:

• The image-only compatibility builder loads the identical vocabulary/template. All 36 prompt combinations match XTuner/HF. Six image geometries match HF pixels/grids exactly. Real single/multiple image records match IDs, modality IDs, pixels and grids; XTuner adds its explicit terminal SFT EOS (2,787 / 16,828 tokens versus 2,786 / 16,827).
• Actual PreTokenizedDatasetWrapper inputs and the generic collator causal shift were exercised. Labels are intentionally not claimed identical: XTuner respects loss=False and supervises reasoning, closing think and stop targets. AM's generic label builder fails to derive a final stop marker from this template and falls back to answer-text matching. It omits reasoning/closing-think supervision and does not consume XTuner's loss annotation. A reproduced user="Answer.", assistant="Answer." case labels user positions 10–11 and masks assistant positions 15–16. This path must not be used as a GLM loss-mask oracle.
• The AM example rejects/replaces overlong samples; XTuner accepts truncation only if media survives intact. These are different training data policies. A controlled four-frame AM/HF processor probe matched, but the AM builder explicitly targets images only, so no AM video training equivalence is claimed.

The probe bypassed only the eager GLM model-package initializer because this shared Torch lacks microbatch.BlockMask; processor, dataset and collator implementations were unmodified. No full AM recipe or GPU run. Evidence: validation/train_infra/multimodal_input_contract_20261014/automodel_comparison/. The follow-up review also corrects guard typing; it does not change the compared token/label output.

Test Plan

CPU, shared pt29_glm2, Transformers 5.17.0, published GLM-5.3-Flash-25B tokenizer/processor assets:

python -m pytest tests/datasets/test_glm53_chat.py \
  tests/datasets/test_glm53_vl_tokenize_fn.py \
  tests/datasets/test_glm53_vl_collator.py -q

• Original upstream suite: 30 passed.
• Independent native-template regressions before the fix: 9 failed, 7 passed; the failures cover the three root causes above.
• Additional boundary/oversized-sample regressions against unmodified upstream F1: all 5 fail as expected.
• Final F1: 55 passed. The full patch applied to latest upstream F6 b5816bd1d: 55 passed. The new oversized-text case preserves text truncation and shifted labels.
• Real single-image and multiple-image records (2,787 and 16,828 tokens): exact full token IDs, modality IDs, pixels and grids; cache/runtime lengths and packed labels/positions also match.
• HF export metadata writer plus tokenizer finalization: reloaded processor reproduces native image generation-prefix inputs exactly. This checks exported assets, not saved model weights or generated answers.
• Ruff check/format and git diff --check: passed. An isolated /tmp pre-commit environment ran the repository hooks: mypy passed all 394 source files, and all formatting/lint hooks passed for the three changed production files. Full pre-commit run --all-files was not run to green locally because the shared environment lacks required modules; the local Pydantic scan imports an unrelated GPU RMSNorm module and fails in this CPU-only run. After the 2026-10-15 rebase onto the force-pushed F1, the PR lint workflow on cc64fc6d1 passes all hooks including docformatter, superseding the earlier inherited docformatter failure. No shared-environment or unrelated-source changes were made.

CPU real single-image tokenize benchmark (one thread, 3 warmup calls excluded, 12 measured calls): cache median 3.871 → 4.038 ms; runtime median 8.124 → 8.155 ms. This excludes initialization and explicit memory trimming and is not GPU throughput data.

No GPU training or deployment-engine generation was run. Mixed image+video within one record remains explicitly unsupported. Draft pending review; the uncovered items are the ones stated above.

Follow-up commits: 25ad5348c (guard typing) and cc64fc6d1 (drops the short-lived cache version prefix; same PR/base, no restack). Validation details: validation/train_infra/multimodal_input_contract_20261014/review_fix/.

CI on cc64fc6d1: the lint workflow passes fully — mypy (394 files), Pydantic, Ruff, docformatter and the other pre-commit hooks (run 38679516704). Both ReadTheDocs builds still fail, as on the F1 base; their root cause has not been established.