A Brown University computer science researcher has published a detailed critique arguing that the prevailing multimodal approach to building artificial general intelligence is fundamentally flawed and unlikely to succeed in the near term. Benjamin Spiegel, a PhD candidate, contends that simply combining large language models with vision and action modules into a single system will not produce the kind of flexible, human-level intelligence that can repair a car, untie a knot, or navigate social situations.
Spiegel's argument, published in The Gradient, cuts against the grain of current industry optimism. While companies race to build bigger models that process text, images, and video together, he warns that this strategy treats the symptoms of narrow intelligence rather than the disease of disembodied reasoning.
Why LLMs Lack a Real World Model
The core of Spiegel's critique targets the assumption that large language models learn implicit models of the physical world simply by predicting the next token in a sequence. He acknowledges that the results can look shockingly human-like. An LLM can describe a kitchen, debate philosophy, or write code in ways that suggest deep understanding.
But Spiegel argues this is largely an illusion created by brute-force memorization of statistical patterns in text. He cites research on OthelloGPT, a transformer trained to predict Othello moves, which appeared to encode a board state in its hidden layers. The catch: Othello is a game of pure symbols. You can play it with pen and paper. Sweeping a floor or driving a car cannot. Spiegel's point is that sequence prediction in a closed symbolic domain does not generalize to the open-ended physics of the real world.
He also notes that generative models can score well on benchmarks by learning idiosyncratic heuristics rather than genuine world models. If a cheaper statistical trick solves the prediction problem, the model will take it. The training objective rewards correlation, not causation.
The Syntax Trap
Spiegel draws on linguistics to make his case sharper. Human language understanding rests on three pillars: syntax (sentence structure), semantics (literal meaning), and pragmatics (context and intent). In humans, these are supported by distinct cognitive systems that fuse together.
LLMs, he suggests, may have collapsed semantics and pragmatics back into syntax. They can generate grammatically perfect sentences with no real-world grounding. A model might correctly parse "The fridge is in the apple" as structurally valid while missing the obvious physical impossibility. Without embodiment, there is no friction to correct such errors. The model learns rules about words, not facts about fridges.
This matters because society increasingly treats chatbot fluency as a proxy for general intelligence. Spiegel warns that this assumption is dangerous when applied to systems that have never touched, moved through, or perceived the world they claim to understand.
The Scale Maximalist Blind Spot
Spiegel directly engages with Rich Sutton's famous "Bitter Lesson," the idea that methods leveraging massive computation eventually outperform those built on human knowledge and structure. He agrees with the lesson's spirit but argues that scale maximalists have misread it.
The success of LLMs and large vision models relied on abundant natural data: the internet's text and image deposits. But AGI requires embodiment data that simply does not exist at comparable scale. You cannot scrape the experience of tying shoelaces or the proprioception of catching a ball from a web crawl.
Multimodal models attempt to bridge this gap by stitching separate modules together into a joint embedding space. Spiegel finds this approach problematic. Meaning becomes decentralized across modality-specific encoders and decoders. The model learns to copy human conceptual structures rather than invent its own. It memorizes the products of intelligence without grasping the process.
What the Alternative Looks Like
Spiegel does not just tear down the current paradigm. He offers a direction: treat embodiment and environmental interaction as primary, and let modality-specific processing emerge naturally. Rather than pre-partitioning perception into "text," "image," and "video" streams, a truly general system might process all sensory input through unified mechanisms and generate all actions through unified motor systems.
He points to his own recent work on visual theory of mind as a small proof of concept. In that research, abstract symbols emerged organically from communication between image-classifying agents, blurring the line between visual and linguistic processing without hard-coding the distinction.
The trade-off is efficiency. Unified, embodied systems will likely train slower and cost more than narrow models tuned for specific tasks. As long as scaling language models solves immediate commercial problems, the economic incentive to pursue this harder path remains weak.
The Road Ahead
Spiegel closes with a striking observation: the hardest mathematical piece of the AGI puzzle, universal function approximation, is already solved. What remains is conceptual architecture. How should the necessary functions be arranged? What kind of world model, value system, and learning mechanism can form new concepts from sparse experience rather than memorizing old ones?
These are not questions that more parameters or more modalities will answer on their own. The AI industry may need to look beyond its current playbook if it wants to build machines that truly think, rather than machines that merely sound like they do.