V3.5.0

Change Log

Feature

• Support StableSPAM optimizer. (#358, #359) * How to Train in 4-Bit More Stably than 16-Bit Adam
• Support ScheduleFreeWrapper. (#334, #360)
• Implement AdaGC optimizer. (#364, #366) * Improving Training Stability for Large Language Model Pretraining
• Implement Simplified-Ademamix optimizer. (#364, #366) * Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants
• Support Ackley function for testing optimization algorithms.

Update

• Update Muon optimizer. (#355, #356) * support decoupled weight decay. * adjust default hyperparameters the same as the original implementation. * support adjusted lr from the Moonlight. you can use it by setting use_adjusted_lr=True.
• Tune the performance of the coupled Newton iteration method by 5% increase. (#360)
• Update SCION optimizer. (#361) * add scale parameter. * update get_lmo_direction.

Fix

• bias_correction2 in ScheduleFreeRAdam optimizer. (#354)
• potential bug in SPAM optimizer. (#365)
• initialize the z state within the step() of the ScheduleFreeWrapper. (#363, #366)