A new architecture, 'Learning to (Learn at Test Time)', has been developed for recurrent neural networks (RNNs), featuring expressive hidden states and linear complexity. This innovation allows the model to 'learn' from its context during test time, replacing the traditional Attention mechanism's costly KV cache. The architecture has shown better perplexity performance compared to Mamba. Developed over the last 1.5 years, the model scales effectively from 125M to 1.3B parameters, demonstrating significant improvements in long-context modeling. The sequence modeling layers were trained from Books scale, further enhancing their performance.