HyperWrite, an AI company, employed A/B testing using Stripe conversion rates to evaluate different language models, prioritizing real-world performance metrics over traditional offline benchmarks. Their tests revealed that GPT-4.1 matched the conversion effectiveness of their existing model, Claude 3.5 Sonnet, while offering substantial cost savings. This approach highlights a shift in model evaluation strategies by focusing on business-relevant outcomes such as customer purchases. The methodology and insights were developed in partnership with OpenAI and detailed in a guide co-authored by HyperWrite's team, emphasizing the importance of selecting evaluation metrics aligned with actual business goals.