V2.7.0

Change Log

Feature

• Implement AdaNorm optimizer (#133)
  • AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs
• Implement RotoGrad optimizer (#124, #134)
  • RotoGrad: Gradient Homogenization in Multitask Learning
• Implement D-Adapt Adan optimizer (#134)
• Support AdaNorm variant (#133, #134)
  • AdaBelief
  • AdamP
  • AdamS
  • AdaPNM
  • diffGrad
  • Lamb
  • RAdam
  • Ranger
  • Adan
• Support AMSGrad variant (#133, #134)
  • diffGrad
  • AdaFactor
• Support degenerated_to_sgd (#133)
  • Ranger
  • Lamb

Refactor

• Rename adamd_debias_term to adam_debias (#133)
• Merge the rectified version with the original (#133)
  • diffRGrad + diffGrad -> diffGrad
  • RaLamb + Lamb -> Lamb
  • now you can simply use with rectify=True

Bug

• Fix previous_grad deepcopy issue in Adan optimizer (#134)

Diff

2.6.1...2.7.0