V3.0.2

Change Log

Feature

• Implement WSD LR Scheduler. (#247, #248)
  • Warmup-Stable-Decay LR Scheduler
• Add more Pytorch built-in lr schedulers. (#248)
• Implement Kate optimizer. (#249, #251)
  • Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad
• Implement StableAdamW optimizer. (#250, #252)
  • Stable and low-precision training for large-scale vision-language models
• Implement AdamMini optimizer. (#246, #253)
  • Use Fewer Learning Rates To Gain More

Refactor

• Refactor Chebyschev lr scheduler modules. (#248)
  • Rename get_chebyshev_lr to get_chebyshev_lr_lambda.
  • Rename get_chebyshev_schedule to get_chebyshev_perm_steps.
  • Call get_chebyshev_schedule function to get LamdbaLR scheduler object.
• Refactor with ScheduleType. (#248)