Recent research has unveiled a counterintuitive finding in the field of artificial intelligence: increasing the number of calls to large language models (LLMs), such as ChatGPT, does not necessarily improve the performance of compound AI systems. This discovery challenges the prevailing assumption that more LLM interactions, through a sample+filter approach, would lead to better outcomes. Researchers have begun to explore the scaling properties of these systems, both theoretically and empirically, to determine the optimal number of LLM calls. The study highlights a non-monotonic relationship between the number of LLM calls and system performance, suggesting that while more calls may benefit simpler tasks, they could hinder performance on more complex problems. Additionally, the research introduces 'tinyBenchmarks', a method for cheap and reliable LLM benchmarking that significantly reduces the need for computing power, by up to 140x, in tasks like MMLU. This insight has significant implications for the development and optimization of AI systems, prompting a reevaluation of strategies for integrating LLMs into compound AI architectures.