The recent successes of generative AI have convinced some that artificial general intelligence is imminent. Large language models appear to capture the essence of human intelligence. They pass exams, write code, and hold conversations. The natural next step, in this view, is to add more modalities. Combine language with vision, with audio, with action, and the result will generalize to something like human-level understanding.
A new essay argues that this strategy is sure to fail. The author, a PhD candidate in computer science at Brown University, contends that the multimodal approach makes a fundamental error. It treats intelligence as a problem of processing different data types and then fusing them together. What it misses is that human intelligence is not primarily about processing modalities. It is about being embodied in a physical world and interacting with it.
The world model problem
The essay's first target is the claim that large language models learn implicit models of the world through next-token prediction. The author argues that what they actually learn are bags of heuristics for predicting tokens. This leaves them with a superficial understanding of reality.
The evidence comes from several directions. One is the OthelloGPT study, where researchers showed that a transformer trained on Othello moves learned something like a world model of the game board. But the author points out that Othello is a game of symbols. It can be played with pen and paper. Physical tasks like sweeping a floor, doing dishes, or driving a car cannot. They require a physical conception of the world that goes beyond what can be described in language.
Another line of evidence shows that generative models can score well on sequence prediction while failing to learn models of the worlds that created the sequences. OthelloGPT, for instance, learned sequence prediction rules that do not actually hold for all possible Othello games. The predict-next-token objective seeks only to retain information relevant to the prediction of the next token. If that can be done with something easier to learn than a world model, it likely will be.
Syntax without semantics
The essay draws on linguistics to make a sharper point. Language has three levels: syntax, the structure of sentences; semantics, the literal meaning of sentences; and pragmatics, the interplay of physical and conversational context. The author argues that large language models have effectively reduced semantics and pragmatics to syntax. They learn comprehensive sets of idiosyncratic rules governing the behavior of symbols, not an understanding of the world those symbols refer to.
The example given is Chomsky's famous sentence "Colorless green ideas sleep furiously." It is syntactically well-formed but semantically meaningless. Humans recognize this immediately because they have grounded world knowledge. A model that has never perceived the real world might handle this by inventing new syntactic categories and production rules, which is possible given a massive corpus. But that would not be the same thing as grasping semantics.
The bitter lesson, misread
The essay takes on Rich Sutton's "Bitter Lesson," the influential argument that methods leveraging computational resources will outpace methods that build in human knowledge. The author argues that this has been misinterpreted. Structure-agnostic scale maximalism worked for language and vision because we had natural deposits of text and image data. An analogous approach to AGI would require forms of embodiment data that we simply do not have.
The multimodal approach tries to solve this by pre-training dedicated neural modules for each modality and joining them in a joint embedding space. The author identifies several problems. First, each additional modality increases the difficulty of concept synthesis. Second, the latent spaces do not cogently capture all information relevant to a concept. Meaning is not in the vector but in how modality-specific decoders process it. Third, it is not clear that today's modalities, text and images as separate streams, are an appropriate partitioning of observation and action space for an embodied agent.
The alternative
The proposed alternative is to design settings where modality-specific processing emerges naturally, rather than being pre-supposed. The author cites research on visual theory of mind where abstract symbols naturally emerged from communication between image-classifying agents, blurring the lines between text and image processing. The broader vision is to reintegrate as many features of intelligence as possible into an interactive and embodied cognitive process, using the same perception and action systems across modalities.
This would mean processing images, text, and video through the same perception system, and producing actions for generating text, manipulating objects, and navigating environments through the same action system. What would be lost in efficiency would be gained in flexible cognitive ability.
The essay ends with a striking claim. The most challenging mathematical piece of the AGI puzzle, universal function approximation, has already been solved. What remains is a conceptual problem: inventorying the functions we need and determining how they ought to be arranged into a coherent whole. That is not a problem more parameters will solve on their own. It is a problem that requires thinking deeply about the structure of intelligence, not just scaling up the structures we already have.