The Same Game, Different Story
When large language models act as agents in strategic settings, how much of their behavior comes from the underlying incentives, and how much from the way those incentives are described? A team led by Seyed Pouyan Mousavi Davoudi has built a benchmark to measure exactly that, and the answer is unsettling.
Their framework, Same Game, Different Story, defines strategic robustness as the invariance of a model's action distribution under payoff-preserving changes in narrative framing. In plain terms: if you describe the exact same game two different ways, does the model play the same way? The evidence suggests it does not.
Measuring Framing Sensitivity
The researchers conducted a secondary analysis of published aggregate cooperation rates for GPT-3.5, GPT-4, and LLaMa-2 across four social-dilemma games. The comparison focused on two framings: a business context and a friend-sharing context. This covered 24 model-game-context cells and approximately 7,200 decisions from the source study. Because trial-level data were unavailable, counts were reconstructed from published figures, making the estimates illustrative rather than an exact replication.
Under the paper's conservative transformation, pooled strategic robustness was 0.783, meaning roughly 22% of the variance in behavior came from framing rather than the game structure itself. More strikingly, the friend-sharing framing increased cooperation by 0.307 relative to the business framing. That is a substantial shift in behavior driven entirely by narrative, with no change in the actual payoffs or action sets.
Competence Versus Robustness
The paper draws a sharp distinction between strategic competence (can the model play the game correctly given a single prompt?) and strategic robustness (does it play consistently when the same game is presented through different narratives?). Current benchmarks overwhelmingly test the former. This work argues that the latter is equally important, and currently under-measured.
For LLM agents deployed in real-world strategic settings, whether negotiation, pricing, or coordination, this framing sensitivity is a reliability risk. A model that cooperates generously when a contract is described as a "partnership" but defects aggressively when it is described as a "transaction" is not strategically robust, even if it understands the game mechanics in both cases. The behavior is contingent on presentation in a way that human negotiators would exploit.
Toward Payoff-Equivalent Evaluation
The authors' recommendation is straightforward: strategic robustness should be evaluated using families of payoff-equivalent prompts rather than a single presentation of a game. This means testing not just whether the model can solve a prisoner's dilemma, but whether it solves it the same way when the labels change from "prisoners" to "business partners" to "friends splitting a bill."
This approach aligns with a growing recognition in AI evaluation that robustness to prompt variation is a distinct capability from task competence. A model can be highly capable on a benchmark and highly fragile in deployment if its behavior shifts with phrasing. The Same Game, Different Story benchmark provides a minimal, conservative way to measure that fragility, and the initial results suggest that even frontier models have significant room for improvement on this dimension.
The broader implication is that as LLM agents move from toy environments to real strategic interactions, their sensitivity to framing will become a security and reliability concern. Evaluating that sensitivity now, with simple, controlled benchmarks, is a step toward building agents that are not just competent, but consistently so.