Repository navigation
Fix custom tokens on Gemma3n/Gemma4 and the Gemma3 tool end token - #819
Open
Nas01010101 wants to merge 2 commits into
Open
Nas01010101 wants to merge 2 commits into
Nas01010101 wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #817. Fixes #818.
Two commits:
_add_custom_tokensstill checksCUSTOM + ifirst. If that piece isn't<unused{i}>, it looks the piece up by name. The name lookup builds a dict over the vocab only on that miss, so Gemma2 and Gemma3 do no extra work. This commit also addsCUSTOM = 6to_Gemma4SpecialTokens, and fixes the "voab" typo in the error message.BEGIN_OF_TOOL_RESPONSE = 50from_Gemma3SpecialTokens(id 50 is<unused44>), andSampleronly adds the token toend_tokenswhen the tokenizer defines it. The same guard also fixes the Gemma2AttributeErrorfrom Issue: Gemma2 sampling crashes with AttributeError when Sampler builds end_tokens (BEGIN_OF_TOOL_RESPONSE) #568, which overlaps with Gemma2 sampling crashes with AttributeError when Sampler buildsend_tokens(BEGIN_OF_TOOL_RESPONSE) (#568) #569, fix: guard BEGIN_OF_TOOL_RESPONSE for Gemma2 and fix f-string in _normalize_token #590 and Support non-tool use models with Sampler #707.Tests
_tokenizer_test.py: addedtest_gemma3_custom_tokens,test_gemma3n_custom_tokensandtest_gemma4_custom_tokens._sampler_test.py: addedtest_end_tokens_exclude_custom_tokens(Gemma3 withcustom_tokens={44: ...}) andtest_end_tokens_gemma2.pytestcommand locally (same--ignores). On main: 392 passed, 3 failed. On this branch: 397 passed, with the same 3 failures (peft/_lora_test.pyand_quantization_test.py,jax._src.prngmissing in my local jax).