This means that this 10M token context is virtual. Kind of "you can try to use it, but beyond 256K tokens, you are on your own," and even below 256K tokens, you are mostly on your own because getting many high-quality examples of such long problems and solutions is hard to https://t.co/LB6Scl0zrs
10M context!? Oooooollama!
10M context - wonder how it compares with @magicailabs https://t.co/Nxgy3HJTKZ
Meta has reportedly set new standards in the AI industry by achieving a 10 million token context window, which can operate on a single GPU and is natively multimodal. This development allows for the processing of extensive data sets, such as 500k-1mil line codebases or large volumes of text, within a single prompt. The context window's capability is highlighted by its potential to handle content equivalent to the entire Harry Potter series, all Slack DMs from 2016, and still have room for 'War and Peace'. It's noted that this 10M token context is virtual, suggesting that beyond 256K tokens, users are largely on their own due to the challenges in obtaining high-quality examples of such long problems and solutions.
