LLMs Process Up to 10M Timesteps, RMT Outperforms GPT-4 with 137M Parameters

Recent advancements in Large Language Models (LLMs) highlight a growing focus on enhancing their capabilities in processing long contexts and generating synthetic data. A novel approach involves fine-tuning GPT-2 with recurrent memory augmentations, enabling it to handle tasks up to 10M timesteps, addressing the challenge of processing long documents. This development is complemented by the introduction of BABILong, a new benchmark for assessing model performance in long document processing. Additionally, tools like DataDreamer are being utilized for synthetic data generation and reproducible LLM workflows, while LLM Comparator offers visual analytics for side-by-side LLM evaluation. Innovations in subquadratic architectures, such as Linear Transformers with learnable kernel functions, are also being explored to improve in-context learning. Moreover, the Recurrent Memory Transformer (RMT), based on GPT2 with 137M parameters, has shown superior performance over GPT-4 and RAG in contexts up to 128K and even 10M, indicating a significant leap in LLMs' ability to manage extensive context lengths.