Sources
Loading...
Additional media
Loading...
DeepNewz, mobile.
People-sourced. AI-powered. Unbiased News.
Jina AI has introduced a new chunking method called 'Late Chunking,' which enhances text embeddings for retrieval tasks by encoding entire documents before splitting them. This technique allows for better retention of context and improved retrieval performance, akin to ColBERT but with the same storage costs as regular methods. The method is available for use within Weaviate with minimal changes to existing code. The research team includes M Günther, I Mohr, B Wang, and H Xiao.
Naive chunking (aka normal mainstream chunking technique) breaks a document into smaller sections before embedding each chunk separately. Although this generally works well, there’s definitely room for improvement. Late chunking is a super interesting retrieval technique… https://t.co/Yf58hTnZjx
Prompt Refinement or Fine-tuning? Best Practices for using LLMs in Computational Social Science Tasks @bimedotcom @Khulood_Almani @theomitsa @FmFrancoise @sulefati7 @NathaliaLeHen @IanLJones98 @sallyeaves @BetaMoroney @sonu_monika @TheAIObserverX https://t.co/YuKHqjSHqi https://t.co/tLvjOicu5F
[CL] Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models M Günther, I Mohr, B Wang, H Xiao [Jina AI GmbH] (2024) https://t.co/QNMcSfuHjN https://t.co/BIBYHInWUo
