Apr 9, 02:44 PM

2024 AI Breakthrough: 70B SoTA LLMs Near Human-Level, Store 2 Bits/Parameter

Recent studies and discussions in the field of artificial intelligence have highlighted significant advancements in the capabilities of Large Language Models (LLMs). Researchers, including Beren Millidge, have pointed out that the parameter counts of human-level vision and language models are closer than previously thought, with current state-of-the-art models comprising 70 billion (70B SoTA) parameters, closely approximating the majority of human capabilities. Further investigations into the knowledge capacity of these models, as conducted by Z Allen-Zhu, Y Li of Meta & Mohamed bin Zayed University of AI (2024), have established that LLMs can store 2 bits of knowledge per parameter, even when quantized to int8, with around 1000 exposures. This finding was corroborated across multiple controlled datasets, indicating that even a 7 billion parameter model can effectively store 14 billion bits of knowledge. The studies also delve into how the size, training, architecture, and data quality of these models affect their capacity to store and utilize knowledge for downstream applications. Additionally, there is an observed trend where the persuasive capabilities of these models are increasing in correlation with their size, suggesting that they are becoming as persuasive as human writers without showing signs of plateauing.

#Large Language Models #Beren Millidge #Meta

Written with ChatGPT (GPT-4).

Sources

Additional media

Image #1 for story 2024-ai-breakthrough-70b-sota-llms-near-human-level-store-2

2024 AI Breakthrough: 70B SoTA LLMs Near Human-Level, Store 2 Bits/Parameter

Sources

Additional media

Similar Stories