This essay argues that rational people don't have goals, and that rational AIs shouldn't have goals. Human actions are rational not because we direct them at some final 'goals,' but because we align actions to practices: networks of actions, action-dispositions, action-evaluation criteria, and action-resources that structure, clarify, develop, and promote themselves. If we want AIs that can genuinely support, collaborate with, or even co-evolve with human agency, AI agents' deliberations must share a "type signature" with the practices-based logic we use to reflect and act.
Eudaimonic Rationality vs. Consequentialist Optimization
The concept of eudaimonia — active, rational human flourishing — doesn't simply point to a desired state of the world that we should set as an AI's optimization target. Rather, it points to a structure of deliberation different from standard consequentialist rationality. This form of rational activity and valuing, called eudaimonic rationality, is a useful or even necessary framework for the agency and values of human-aligned AIs.
In a eudaimonic model of rational activity, a rational action is an element of a valued practice in roughly the same sense that a note is an element of a melody, a time-step is an element of a computation, and a moment in an organism's cellular life is an element of that organism's self-subsistence and self-development. There is no strict distinction between means and ends, or between 'instrumental' and 'terminal' values.
The Material Efficacy Condition
For a practice to be fit for possessing internal criteria of flourishing, excellence, and eudaimonic rationality, a practice must materially allow for an optimally self-promoting property that strongly correlates with a plethora of more local, more individually measurable properties whose instantiation is prima facie valuable. Stated more informally, there must exist a two-way causal relationship between a practice's excellence and the material, psychological, and epistemic effects of its excellence, such that present excellence reliably materially, psychologically, and epistemically promotes future excellence.
Implications for AI Alignment
Many puzzles and 'paradoxes' about AI alignment are driven by the assumption that mature AI agents will be Effective Altruism-style optimizers. A "type mismatch" between Effective Altruism-style optimization and eudaimonic rationality makes it nearly impossible to translate the interests of humans — agents who practice eudaimonic rationality — into a utility function legible to an Effective Altruism-style optimizer AI.
Conceiving of corrigibility, transparency, or niceness as adverbial practices is a promising way to capture the normal, sensible way we want an agent to value these properties — which intuitively consequentialist values and deontology both fail to capture. We want an agent that actively tries to be transparent, and to cultivate its own future transparency, but that will not engage in deception and plotting when it expects a high future-transparency payoff.
What makes human life beautiful is also what makes human life possible at all. If this is right, then eudaimonic rationality is not a matter of congratulating ourselves for our richly human ways of reasoning, valuing, and acting but a key to basic sanity.