V2.9.0

Change Log

Feature

• Implement AdaMax optimizer (#148)
  • A variant of Adam based on the infinity norm
• Implement Gravity optimizer (#151)
  • a Kinematic Approach on Optimization in Deep Learning
• Implement AdaSmooth optimizer (#153)
  • An Adaptive Learning Rate Method based on Effective Ratio
• Implement SRMM optimizer (#154)
  • Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates
• Implement AvaGrad optimizer (#155)
  • Domain-independent Dominance of Adaptive Methods
• Implement AdaShift optimizer (#157)
  • Decorrelation and Convergence of Adaptive Learning Rate Methods
• Upgrade to D-Adaptation v3 (#158, #159)
• Implement AdaDelta optimizer (#160)
  • An Adaptive Learning Rate Method

Docs

• Fix readthedocs build issue (#156)
• Move citations into table (#156)

Refactor

• Refactor validation logic (#149, #150)
• Rename amsbound, amsgrad terms into ams_bound (#149)
• Return gradient instead of the parameter, AGC. (#149)
• Refactor duplicates (e.g. rectified step size, AMSBound, AdamD, AdaNorm, weight decay) into re-usable functions (#150)
• Move pytorch_optimizer.experimental under pytorch_optimizer.*.experimental

Diff

2.8.0...2.9.0