V2.10.0

Change Log

Implement Amos optimizer (#174)
- An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale
Implement SignSGD optimizer (#176) (thanks to @i404788)
- Compressed Optimisation for Non-Convex Problems
Implement AdaHessian optimizer (#176) (thanks to @i404788)
- An Adaptive Second Order Optimizer for Machine Learning
Implement SophiaH optimizer (#173, #176) (thanks to @i404788)
- A Scalable Stochastic Second-order Optimizer for Language Model Pre-training
Implement re-usable functions to compute hessian in BaseOptimizer (#176, #177) (thanks to @i404788)
- two types of distribution are supported (gaussian, rademacher).
Support AdamD variant for AdaHessian optimizer (#177)