V3.0.0

Change Log

Major version is updated! (v2.12.0 -> v3.0.0) (#164)

Feature

• Implement REX lr scheduler. (#217, #222)
  • Revisiting Budgeted Training with an Improved Schedule
• Implement Aida optimizer. (#220, #221)
  • A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
• Implement WSAM optimizer. (#213, #216)
  • Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term
• Implement GaLore optimizer. (#224, #228)
  • Memory-Efficient LLM Training by Gradient Low-Rank Projection
• Implement Adalite optimizer. (#225, #229)

Fix

• Fix SRMM to allow operation beyond memory_length. (#227)

Dependency

• Drop Python 3.7 support officially. (#221)
  • Please check the README.
• Update bitsandbytes to 0.43.0. (#228)

Docs

• Add missing parameters in Ranger21 optimizer document. (#214, #215)
• Fix WSAM optimizer paper link. (#219)

Contributions

thanks to @sdbds, @i404788

Diff

2.12.0...3.0.0