Skip to content

Exclude parameters marked with _no_weight_decay from weight decay - #55

Open
flyingdoubleG wants to merge 1 commit into
fla-org:mainfrom
flyingdoubleG:no-weight-decay
Open

flyingdoubleG wants to merge 1 commit into
fla-org:mainfrom
flyingdoubleG:no-weight-decay

Conversation

@flyingdoubleG

Copy link
Copy Markdown

Fixes #45.

FLA models mark parameters such as A_log, D and dt_bias with _no_weight_decay, but the torchtitan optimizers put every parameter into one group, so these get decayed like everything else.

This adds --optimizer.respect_no_weight_decay. With it, flame/components/optimizer.py builds the same containers as torchtitan's build_optimizers (plain, in-backward and FT), but with the marked parameters in a second param group with weight_decay=0.

FSDP replaces the parameter objects, so an attribute set in a layer's __init__ does not survive parallelize_fn. The trainer therefore records the names of the marked parameters right before parallelize_fn (record_no_weight_decay), and the builder excludes a parameter if it is in that set (ignoring checkpoint/compile/FSDP wrapper prefixes) or still carries the attribute, e.g. when _init_weights sets it after parallelization. Recording right before parallelize_fn also stays correct if post_init moves before parallelization as in #50.

The flag is off by default: the extra param group changes the optimizer state, so a checkpoint saved without the flag can't be resumed with it and vice versa. Happy to flip the default if you prefer.

Tests (tests/test_optimizer.py): the marked parameters are untouched by a step with zero gradients while the others decay; names still match after the parameter is replaced and the module wrapped; build_optimizers with the flag on and off (skipped when torchtitan isn't installed, as in CI). Run locally with torchtitan@0b44d4c on CPU; I haven't run a multi-GPU FSDP job with it.

With --optimizer.respect_no_weight_decay, the optimizers put the parameters FLA models mark with _no_weight_decay (A_log, D, dt_bias, ...) into a parameter group without weight decay. The marked names are recorded before parallelisms replace the parameters. Off by default, since the extra parameter group changes the optimizer state of checkpoints. Fixes fla-org#45.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Read the ._no_weight_decay attribute of parameters

1 participant