ICML2024: JinaAI's CLIP Model Improves Text Encoder by 165%, New Lumina-mGPT Model for Vision and Language Tasks

Recent advancements in multimodal generative models have been highlighted in various presentations and papers. At the #ICML2024 conference, a significant improvement in the text encoder performance of JinaAI's CLIP model was reported, achieving a 165% increase in capability. This was detailed by a colleague of the presenter, emphasizing the effectiveness of contrastive text-image pre-training for cross-modal retrieval, although it noted limitations in text capability. Additionally, the introduction of Lumina-mGPT was announced, a family of multimodal autoregressive models designed for diverse vision and language tasks. Other discussions included the benefits of adapting large pre-trained text-to-image models for immersive scene generation, particularly in panorama creation, which addresses the challenges posed by the high costs of acquiring multi-view images. Despite these advancements, it was noted that existing vision-language models still struggle with reasoning over spatial relationships. The TexGen project was also presented, focusing on text-guided 3D texture generation through multi-view sampling and resampling techniques.