V3.10.1

Change Log

Feature

• Implement LoRARite optimizer. (#498, #504)
  • LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
• Implement FlashAdamW optimizer. (#495, #503)
  • FlashOptim: Optimizers for Memory-Efficient Training
• Implement DualAdam optimizer. (#496, #502)
  • Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers
• Implement ROSE optimizer. (#500, #501)
  • Stateless optimization through range-normalized gradient updates

Contributions

thanks to @mfazrinizar