Apple has introduced the AdEMAMix Optimizer, a novel Adam-based optimizer that leverages very old gradients to achieve better solutions. This optimizer has been tested on models such as Transformer LM, Mamba LM, and Vision Transformer (ViT) training. Notably, the 1.3 billion parameter AdEMAMix Transformer language model (LM) trained on 101 billion tokens performs comparably to the AdamW optimizer but requires 95% fewer training tokens to reach the same loss.