A World in Which the Alignment Problem Seems Lower-Stakes

The danger from power-seeking is not intrinsic to the alignment problem. This danger also depends on the structure of the agent’s environment.

In The Catastrophic Convergence Conjecture, I wrote:

Are there worlds where this isn’t true? Consider a world where you supply a utility-maximizing agi with a utility function.

The only information you have about the other half is your utility. For simplicity, let’s suppose you and the agi have utility functions over universe-histories which are additive across the halves of the universe. You don’t observe any utility information about the other part of the universe until the end of time, and vice versa for the agi. That is, for history $h$,

If the agi uses something like causal decision theory, then it won’t try to kill you, or “seek power” over you. The effects of its actions have no causal influence over what happens in your half of the universe. Your universe’s evolution adds a constant term to its expected utility.

Note

Other decision theories might have the agi precommit to minimizing human utility unless it attains maximal agi-utility from the left half of the universe-history, or some other shenanigans. This precommitment isn’t relevant to the point I want to make in this post, but it’s important to consider.

However, the setup is still interesting because

1. Goodhart’s law still applies: if you give the agi an incomplete proxy objective, you’ll get suboptimal true performance.
2. Value is still complex: it’s still hard to get the agi to optimize the right half of the universe for human flourishing.
3. If the agi is autonomously trained via stochastic gradient descent in the right half of the universe, then we may still hit inner alignment problems.

Alignment is still hard, and we still want to get the agi to do good things on its half of the universe. But it isn’t instrumentally convergent for the agi to seek power over you, and so you shouldn’t expect an unaligned agi to try to kill you in this universe. You shouldn’t expect the agi to kill other humans, either, since none exist in the right half of the universe—and it won’t create any, either.

To restate: Bostrom’s original instrumental convergence thesis needs to be applied carefully. The danger from power-seeking is not intrinsic to the alignment problem. This danger also depends on the structure of the agent’s environment. I think I sometimes bump into reasoning that feels like “instrumental convergence, smart AI, & humans exist in the universe → bad things happen to us / the AI finds a way to hurt us”; I think this is usually true, but not necessarily true, and so this extreme example illustrates how the implication can fail.

Thanks

Thanks to John Wentworth for feedback on this post. Edited to clarify the broader point I’m making.

☙❧