Concerns Raised Over GSM8K's 99.2% Accuracy Amid Data Leakage Fears

Recent discussions on social media have raised concerns about the validity of the GSM8K evaluation metric, which reportedly has achieved a 99.2% accuracy rate. Some experts believe that this high accuracy may be indicative of data leakage and overfitting. Lukasz Kaiser expressed skepticism about the accuracy, finding it hard to believe that only 1% of the test examples are questionable. Another user suggested that the 99.2% accuracy might actually be a bad signal, while Swaroop Rm7 called for a robustness study involving paraphrasing questions or changing numbers to verify the results.