V3.6.1

Change Log

Feature

• Implement more cooldown types for WSD learning rate scheduler. (#382, #386)
• Implement AdamWSN optimizer. (#387, #389) * Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees
• Implement AdamC optimizer. (#388, #390) * Why Gradients Rapidly Increase Near the End of Training

Update

• Change the default range of the beta parameter from [0, 1] to [0, 1). (#392)

Fix

• Fix to use momentum buffer instead of the gradient to calculate LMO. (#385)