
Recent advancements in local language models (LLMs) have significantly enhanced their capabilities and accessibility. The latest open-source Llama 3.2 model can now be used to chat with PDF files through a Retrieval-Augmented Generation (RAG) approach. This model, along with others like Qwen 2.5 72B and Llama 3.1 70B, can be run locally on devices such as Apple Silicon Macs, ensuring privacy and offline functionality. The LM Studio 0.3.4 update, which includes Apple MLX, allows Llama 3.2 to run at approximately 250 tokens per second on the new M3 chip. Additionally, the new SOTA visual document retriever, ColPali, enhances the ability to perform multimodal RAG with complex PDFs and slide decks. The update also supports structured JSON responses and integrates ColQwen2 for improved performance.
