fix(trainer): correct LR schedule, final checkpoint and STG under DDP - #263
Open
blue (Blueyyds) wants to merge 1 commit into
Open
fix(trainer): correct LR schedule, final checkpoint and STG under DDP#263blue (Blueyyds) wants to merge 1 commit into
blue (Blueyyds) wants to merge 1 commit into
Conversation
Three defects that only surface on multi-GPU runs or with checkpoint retention enabled: 1. LR scheduler advanced once per process. Accelerate's default step_scheduler_with_optimizer=True applies its distributed batch-size adjustment on top of the trainer's own explicit scheduler.step() per optimizer step, so an 8-GPU run consumed the schedule 8x too fast and hit the final LR after 1/8 of training. The trainer owns the stepping, so opt out. 2. Final checkpoint could delete itself. When the last optimizer step also lands on a save interval, the same path was written twice and appended to the retention list twice; with keep_last_n=1 the pruning pass then removed the file it had just written. Skip the redundant save when the interval checkpoint already exists, with a wait_for_everyone() so all ranks agree on the state. 3. STG validation crashed under DDP. transformer.num_blocks is not reachable through the DistributedDataParallel wrapper, so enabling stg_scale raised AttributeError during validation. Read the attribute off .module when wrapped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three defects that only surface on multi-GPU runs or with checkpoint retention enabled:
LR scheduler advanced once per process. Accelerate's default step_scheduler_with_optimizer=True applies its distributed batch-size adjustment on top of the trainer's own explicit scheduler.step() per optimizer step, so an 8-GPU run consumed the schedule 8x too fast and hit the final LR after 1/8 of training. The trainer owns the stepping, so opt out.
Final checkpoint could delete itself. When the last optimizer step also lands on a save interval, the same path was written twice and appended to the retention list twice; with keep_last_n=1 the pruning pass then removed the file it had just written. Skip the redundant save when the interval checkpoint already exists, with a wait_for_everyone() so all ranks agree on the state.
STG validation crashed under DDP. transformer.num_blocks is not reachable through the DistributedDataParallel wrapper, so enabling stg_scale raised AttributeError during validation. Read the attribute off .module when wrapped.