Recent studies have highlighted the limitations of Large Language Models (LLMs) in performing mathematical tasks. A new paper titled 'Are Large Language Models Solving Math?' reveals that LLMs often rely on heuristics and pattern matching rather than actual computation, suggesting that their apparent mathematical abilities may be more guesswork than genuine problem-solving. The authors conclude that enhancing LLMs' mathematical capabilities could necessitate fundamental changes to their training and architectures, rather than merely applying post-hoc techniques. Additionally, the introduction of dynamic puzzle generation benchmarks has exposed the reasoning capabilities of LLMs, indicating a complex interplay between memorization and reasoning in logical problem-solving. Despite their impressive feats in generating human-like text and performing various tasks, LLMs still face significant challenges in complex reasoning and planning due to their fixed computational budgets.