
Meta has introduced a novel training method called Reverse Training aimed at addressing a significant challenge in large language models (LLMs) known as the 'Reversal Curse'. This issue arises when LLMs, trained on statements like 'A has a feature B', fail to generalize to the inverse 'B is a feature of A'. Reverse Training seeks to remedy this by doubling the training tokens through the reversal of strings, thereby enhancing the models' ability to handle reversal tasks. This method not only outperforms data-matched standard baselines but also fixes issues related to the Reversal Curse, marking a significant advancement in the field of LLMs. The research, conducted by O Golovneva, Z Allen-Zhu, J Weston, S Sukhbaatar at FAIR at Meta in 2024, introduces reverse training as a strategy for LLMs to combat the Reversal Curse by doubling datasets and preserving entities for improved bidirectional knowledge.
Introducing reverse training as a strategy for LLMs to combat the "Reversal Curse" by doubling datasets & preserving entities for improved bidirectional knowledge: https://t.co/uttkFRA8DJ https://t.co/hHhJVthozR
[CL] Reverse Training to Nurse the Reversal Curse O Golovneva, Z Allen-Zhu, J Weston, S Sukhbaatar [FAIR at Meta] (2024) https://t.co/EMoYLzsTwG - LLMs suffer from the reversal curse: they cannot generalize a fact like "A has feature B" to answering questions about "B is feature… https://t.co/VajsLkYVjf
Meta announces Reverse Training to Nurse the Reversal Curse Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of https://t.co/HumuA6CCHX
