V2.8.0

Change Log

Feature

• Implement A2Grad optimizer (#136)
  • Optimal Adaptive and Accelerated Stochastic Gradient Descent
• Implement Accelerated SGD optimizer (#137)
  • Accelerating Stochastic Gradient Descent For Least Squares Regression
• Implement Adaptive SGD optimizer (#139)
  • Adaptive Gradient Descent without Descent
• Implement SGDW optimizer (#139)
  • Decoupled Weight Decay Regularization
• Implement Yogi optimizer (#140)
  • Adaptive Methods for Nonconvex Optimization
• Implement SWATS optimizer (#141)
  • Improving Generalization Performance by Switching from Adam to SGD
• Implement Fromage optimizer (#142)
  • On the distance between two neural networks and the stability of learning
• Implement MSVAG optimizer (#143)
  • Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients
• Implement AdaMod optimizer (#144)
  • An Adaptive and Momental Bound Method for Stochastic Learning
• Implement AggMo optimizer (#145)
  • Aggregated Momentum: Stability Through Passive Damping
• Implement QHAdam, QHM optimizers (#146)
  • Quasi-hyperbolic momentum and Adam for deep learning
• Implement PID optimizer (#147)
  • A PID Controller Approach for Stochastic Optimization of Deep Networks

Bug

• Fix update in Lion optimizer (#135)
• Fix momentum_buffer in SGDP optimizer (#139)

Diff

2.7.0...2.8.0