Apple Researchers Reveal Structural Failures in Large Reasoning Models Using Puzzle-Based Evaluation

Artificial intelligence has evolved from basic language models to advanced systems known as Large Reasoning Models (LRMs). These tools aim to simulate human-like thinking by generating intermediate reasoning steps before arriving at conclusions. This transition has raised critical questions about how these models handle complex tasks and whether they genuinely possess reasoning abilities or merely rely on learned patterns to produce outcomes.

Evaluating Reasoning: Moving Beyond Final Answer Accuracy

A significant challenge in machine reasoning evaluation is that traditional benchmarks assess only the final answer, ignoring the process of reaching that conclusion. This focus fails to reveal the quality of the internal reasoning and can provide a skewed understanding of a model’s capabilities, especially if benchmark data overlaps with training datasets. To gain insights into actual reasoning, researchers need environments where problem complexity can be precisely managed, and intermediate steps can be thoroughly analyzed.

The research team at Apple designed a comparative study featuring four puzzle environments: Tower of Hanoi, River Crossing, Checkers Jumping, and Blocks World. These settings enable precise complexity manipulation by varying the number of disks, checkers, or agents involved. Each task demands different reasoning capabilities, such as constraint satisfaction and sequential planning, and minimizes data contamination risks while facilitating detailed outcome and reasoning step assessment.

Comparative Insights: Thinking vs. Non-Thinking Models Under Stress

The study employed two sets of models: Claude 3.7 Sonnet and DeepSeek-R1, including their thinking variants and standard LLM counterparts. The models were assessed across the puzzles under identical token budgets to quantify both accuracy and reasoning efficiency. Observing performance across complexities revealed three performance zones. In simpler tasks, non-thinking models performed better, whereas reasoning models excelled in medium-complexity tasks. However, both types faltered in high-complexity scenarios.

The analysis showed reasoned effort increased with task difficulty to a point but then dropped despite plentiful resources. Notably, Claude 3.7 Sonnet (thinking) showed high accuracy in the Tower of Hanoi up to a certain complexity threshold but dropped to zero beyond that. Even when provided with explicit algorithms, models failed to manage simple tasks when complexity levels surged. This inconsistency highlighted significant issues in symbolic manipulation and precise computation.

Scaling Limits and the Collapse of Reasoning

This research by Apple emphasizes the limitations of current LRMs. Despite advancements, these models remain inadequate at achieving generalized reasoning. The study identifies performance scaling, collapse points, and illustrates how over-reliance on benchmark accuracy does not capture essential reasoning behaviors. These controlled puzzle environments have proven effective in exposing underlying weaknesses in LRM designs, underscoring the necessity for more resilient systems in future AI developments.

For more detailed insights, check out the original research paper. All credit for this study goes to the researchers involved.