On September 5th, Matt Shumer announced Reflection 70B, a model fine-tuned on top of Llama 3.1 70B, which showed state-of-the-art benchmark numbers. The model was trained using Glaive generated data. On October 2, Sahil released model artifacts to reproduce the initial claims and provided a post-mortem report addressing various issues. The open-source AI community has been actively examining the model's performance on benchmarks such as HumanEval, GPQA, and MMLU. Despite some evaluation bugs and rushed research code, the community remains optimistic about the model's potential. On October 3, Matt Shumer stated he is still validating the model and is encouraged by the transparency shown in the post-mortem report.