
Recent advancements in text generation technology have been highlighted with the introduction of dynamic speculative decoding by Intel, which accelerates the process by 2-3 times. This method involves a two-stage generative process, utilizing a smaller, less accurate draft model. Additionally, the DISCO framework enhances speculative decoding speed by 10-100%, now set as the default in the Transformers library for assisted generation. The scaling law for large language models (LLMs) indicates that increased compute and data lead to improved performance, a principle that holds true during inference time as well. New experiments suggest that Superposed Decoding can significantly assist in scaling inference time without raising compute costs. These developments will be further discussed at the upcoming NeurIPS Conference in Vancouver.



