V3.3.3

Change Log

Feature

• Implement Grams optimizer. (#317, #318) * Grams: Gradient Descent with Adaptive Momentum Scaling
• Support stable_adamw variant for ADOPT and AdEMAMix optimizer. (#321) * optimizer = ADOPT(model.parameters(), ..., stable_adamw=True)
• Implement an experimental optimizer Ranger25 (not tested). (#321) * mixing ADOPT + AdEMAMix + StableAdamW + Cautious + RAdam optimizers.
• Implement OrthoGrad optimizer. (#321) * Grokking at the Edge of Numerical Stability
• Support Adam-Atan2 feature for Prodigy optimizer when eps is None. (#321) * Scaling Exponents Across Parameterizations and Optimizers