Repository navigation
Exclude parameters marked with _no_weight_decay from weight decay - #55
Open
flyingdoubleG wants to merge 1 commit into
Open
flyingdoubleG wants to merge 1 commit into
flyingdoubleG wants to merge 1 commit into
Conversation
With --optimizer.respect_no_weight_decay, the optimizers put the parameters FLA models mark with _no_weight_decay (A_log, D, dt_bias, ...) into a parameter group without weight decay. The marked names are recorded before parallelisms replace the parameters. Off by default, since the extra parameter group changes the optimizer state of checkpoints. Fixes fla-org#45.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #45.
FLA models mark parameters such as
A_log,Danddt_biaswith_no_weight_decay, but the torchtitan optimizers put every parameter into one group, so these get decayed like everything else.This adds
--optimizer.respect_no_weight_decay. With it,flame/components/optimizer.pybuilds the same containers as torchtitan'sbuild_optimizers(plain, in-backward and FT), but with the marked parameters in a second param group withweight_decay=0.FSDP replaces the parameter objects, so an attribute set in a layer's
__init__does not surviveparallelize_fn. The trainer therefore records the names of the marked parameters right beforeparallelize_fn(record_no_weight_decay), and the builder excludes a parameter if it is in that set (ignoring checkpoint/compile/FSDP wrapper prefixes) or still carries the attribute, e.g. when_init_weightssets it after parallelization. Recording right beforeparallelize_fnalso stays correct ifpost_initmoves before parallelization as in #50.The flag is off by default: the extra param group changes the optimizer state, so a checkpoint saved without the flag can't be resumed with it and vice versa. Happy to flip the default if you prefer.
Tests (
tests/test_optimizer.py): the marked parameters are untouched by a step with zero gradients while the others decay; names still match after the parameter is replaced and the module wrapped;build_optimizerswith the flag on and off (skipped when torchtitan isn't installed, as in CI). Run locally with torchtitan@0b44d4c on CPU; I haven't run a multi-GPU FSDP job with it.