When Smarter Means Narrower
A research team led by Junyi Sha has identified a subtle but consequential failure mode in how large language models learn to make decisions. In deterministic board games based on tic-tac-toe variants, where optimal moves can be computed exactly, the team found that reasoning-mode generation frequently suppresses the range of actions a model considers, even when that narrowing does not translate into better accuracy.
The paper, published on arXiv, frames this as a structural problem in how models imitate behavior during supervised fine-tuning (SFT). When a model is trained on a single demonstrated action per state, it learns to converge on that specific move rather than preserving the full set of valid optimal actions. The result is what the authors call narrow-support imitation: a policy that technically plays well but has lost the exploratory breadth needed for robust sequential decision-making.
The Accuracy-Diversity Trap
Across three evaluation settings, state-level assessment, arena gameplay, and training trajectory analysis, the researchers observed that standard SFT improves action accuracy but often induces premature diversity collapse. This collapse goes beyond what the inherent accuracy-diversity tradeoff would predict. In other words, the model becomes more predictable than it needs to be, shedding valid alternatives before accuracy gains justify the loss.
This matters because diversity in action support is not just a statistical curiosity. In game play and broader reinforcement learning, maintaining multiple viable options per state is what keeps a policy adaptable to opponent variation, distributional shift, and unseen board configurations. A model that always picks the same optimal move when several exist becomes brittle.
Action Augmentation as a Partial Fix
The team tested a straightforward intervention: action augmentation, training the model on every optimal action available in a given state rather than a single demonstration. This approach partially mitigated the collapse, suggesting that the training data format itself shapes the policy's support width. The finding points to a broader design principle for LLM decision-making systems: preserving action support during imitation learning is not optional if the goal is to maintain exploratory behavior.
The authors connect this phenomenon to a wider pattern in LLM training. Policy collapse, where a model's output distribution shrinks to a narrow peak, has been observed in other contexts, from mode collapse in generative models to reward hacking in reinforcement learning from human feedback. What this work adds is a controlled, measurable setting where the effect can be isolated and the remedy tested with precision.
Why This Matters for Agent Design
The implications extend beyond board games. As LLMs are increasingly deployed as agents in strategic, multi-step environments, from coding assistants to negotiation bots, the way they compress their action space during training becomes a reliability concern. A model that reasons its way to a single "correct" move may look competent in static evaluation but fail when the environment demands flexibility. The research suggests that competence and robustness may require different training objectives, and that SFT alone, even with reasoning, may optimize for the former at the expense of the latter.
The paper's core argument is that diversity should be treated as a first-class training target, not an accidental byproduct of accuracy optimization. For practitioners building agentic systems, that means reconsidering how demonstration data is curated and whether the training signal explicitly rewards maintaining a broad, well-calibrated action distribution.