
Chinese research lab GAIR has introduced a new framework called GAIR-ProX, designed to refine pre-training data in large language models (LLMs). This framework, which treats data refinement as a programming task, allows models to perform fine-grained operations such as string normalization. The approach is efficient, enabling even small models with 0.3 billion parameters to refine data with human-like accuracy. ProX achieves significant speedup, up to 20 times faster training in general and math domains, and reduces memory footprint by 88.16%. Additionally, the framework supports collaborative edge AI fine-tuning through the Pluto and Charon (PAC) framework, achieving up to 8.64X speedup and employing parallel adapters to enhance performance.




