The recent successes of generative AI models have convinced some that AGI is imminent. While these models appear to capture the essence of human intelligence, they defy even our most basic intuitions about it. They have emerged not because they are thoughtful solutions to the problem of intelligence, but because they effectively capitalize on scale. Seduced by the fruits of scale, some have come to believe that it provides a clear pathway to AGI. The most emblematic case of this is the multimodal approach, in which massive modular networks are optimized for an array of modalities that, taken together, approximate general intelligence. However, this essay argues that this strategy is sure to fail in the near term; it will not lead to human-level AGI that can perform sensorimotor reasoning, motion planning, and social coordination. Instead of trying to glue modalities together into a patchwork AGI, we should pursue approaches to intelligence that treat embodiment and interaction with the environment as primary, and see modality-centered processing as emergent phenomena.
The world model problem
The essay first argues that many problems require an understanding of the physical world, as many problems originate in physical reality. It has been suggested by some that LLMs are learning a model of the world through next token prediction, but it is more likely that LLMs are learning bags of heuristics to predict tokens. This leaves them with a superficial understanding of reality and contributes to false impressions of their intelligence.
The most shocking result of the predict-next-token objective is that it yields AI models that reflect a deeply human-like understanding of the world, despite having never observed it like we have. This result has sparked debate about what it means to understand and even to mean — something we have long believed to be a prerequisite for language understanding. Proponents of this theory cite the prowess of SOTA LLMs on various benchmarks, the OthelloGPT experiments, and their favorite rendition of the idea that "language mirrors the structure of reality."
One source of evidence in favor of the LLM world modeling hypothesis is the OthelloGPT experiment, wherein researchers were able to predict the board of an Othello game from the hidden states of a transformer model trained on sequences of moves. However, there are issues with generalizing these results to models of natural language. For one, whereas Othello moves can be used to deduce the full state of an Othello board, we have no reason to believe that a complete picture of the physical world can be inferred by a linguistic description. What sets the game of Othello apart from many tasks in the physical world is that Othello fundamentally resides in the land of symbols, and is merely implemented using physical tokens to make it easier for humans to play.
Another issue is that there is evidence that generative models can score remarkably well on sequence prediction tasks while failing to learn models of the worlds that created such sequence data, e.g., by learning comprehensive sets of idiosyncratic heuristics. To claim without caveat that predicting the next token requires a model of the world like the ones humans generate from perception would be to abuse the Goodhart's law. Unless we disagree on what the world is, it should be clear that a world model can be used to predict the next state of the world given a history of states. Similar world models, which predict high fidelity observations of the physical world, are leveraged in many subfields of AI including robotics, autonomous driving, and simulation to solve problems instantiated in physical reality. LLMs are simply not running physics simulations in their latent next-token calculus.
The multimodal critique
The essay argues that Sutton's Bitter Lesson has sometimes been interpreted as meaning that making assumptions about the structure of AI is a mistake. This is both unproductive and a misinterpretation; it is precisely when humans think deeply about the structure of intelligence that major advancements occur. Despite this, scale maximalists have implicitly suggested that multimodal models can be a structure-agnostic framework for AGI. Ironically, today's multimodal models contradict Sutton's Bitter Lesson by making implicit assumptions about the structure of individual modalities and how they should be sewn together. In order to build AGI, we must either think deeply about how to unite existing modalities, or dispense with them altogether in favor of an interactive and embodied cognitive process.
The paradigm that led to the success of LLMs is marked primarily by scale, not efficiency. Despite many decades of structuralist study in linguistics, the unstructured approaches of scale maximalism have yielded far better results in recent years. This was most notably articulated by Rich Sutton in his piece "The Bitter Lesson."
The essay identifies multiple issues with the multimodal approach. First, modalities have vastly different data structures, making the problem of concept synthesis ever more difficult. In practice, uniting modalities often involves pre-training dedicated neural modules for each modality and joining them together into a joint embedding space. While modern approaches do not make such stringent assumptions about how modalities should be united, they still universally encode percepts from all modalities into the same latent space.
Furthermore, it is not clear that today's modalities are an appropriate partitioning of the observation and action spaces for an embodied agent. It is not obvious that images and text should be represented as separate observation streams, nor text production and motion planning as separate action capabilities. The human capacities for reading, seeing, speaking, and moving are ultimately mediated by overlapping cognitive structures. Making structural assumptions about how modalities ought to be processed is likely to hinder the discovery of more fundamental cognition that is responsible for processing data in all modalities.
Finally, the learn-from-scale approach trains models to copy the conceptual structure of humans instead of learning the general capability to form novel concepts on their own. Humans have spent hundreds of thousands of years refining concepts and passing them memetically through culture and language. Today's models are trained only on the end result of this process. The flexibility to form new concepts from experience is a foundational attribute of general intelligence.
A path forward
While structure-agnostic scale maximalism has succeeded in producing LLMs and LVMs that pass Turing tests, a multimodal scale maximalist approach to AGI will not bear similar fruit. Instead of pre-supposing structure in individual modalities, we should design a setting in which modality-specific processing emerges naturally. Instead of trying to glue modalities together into a patchwork AGI, we should pursue approaches to intelligence that treat embodiment and interaction with the environment as primary, and see modality-centered processing as emergent phenomena.
The overall promise of scale maximalism is that a Frankenstein AGI can be sewed together using general models of narrow domains. This essay argues that this is extremely unlikely to yield an AGI that feels complete in its intelligence. If we intend to continue reaping the streamlined efficiency of modality-specific processing, we must be intentional in how modalities are united — ideally drawing from human intuition and classical fields of study. Alternatively, we can re-formulate learning as an embodied and interactive process where disparate modalities naturally fuse together.