AU landscape as a unifying frame

Attainable utilities are calculated by winding your way through possibility-space, considering and discarding possibility after possibility to find the best plan you can. This frame is unifying.

Sometimes you advantage one AU at the cost of another, moving through the state space towards the best possibilities for one goal and away from the best possibilities for another goal. This tradeoff is opportunity cost.

Sometimes you gain more control over the future: most of the best possibilities make use of a windfall of cash. Sometimes you act to preserve control over the future: most Tic-Tac-Toe goals involve not ending the game right away. Otherwise put: preserving power.

Other people usually objectively impact you by decreasing or increasing a bunch of your AUs (generally, by changing your power). This happens for an extremely wide range of goals because of the structure of the environment.

Sometimes, the best possibilities are made unavailable or worsened only for goals much like yours. This outcome involves a value impact for your goals.

Sometimes a bunch of the best possibilities go through the same part of the future: fast travel to random places on Earth usually involves the airport. This commonality is instrumental convergence.

Exercise

Track what’s happening to your various AUs during the following story: you win the lottery. Being an effective spender, you use most of your cash to buy a majority stake in a major logging company. Two months later, the company goes under.

☙❧

Appendix: AU landscape and world state contain equal information

In the context of finite deterministic Markov decision processes, there’s a wonderful handful of theorems which basically say that the AU landscape and the environmental dynamics encode each other. That is, they contain the same information, just with different emphasis. This supports thinking of the AU landscape as a “dual” of the world state.

Quote

Let $⟨\mathcal{S},\mathcal{A},T,\gamma ⟩$ be a rewardless deterministic mdp with finite state and action spaces $\mathcal{S},\mathcal{A}$, deterministic transition function $T$, and discount factor $\gamma \in (0,1)$. As our interest concerns optimal value functions, we consider only stationary, deterministic policies: $\Pi \stackrel{\mathrm{def}}{=}{\mathcal{A}}^{\mathcal{S}}$.

The first key insight is to consider not policies, but the trajectories induced by policies from a given state; to not look at the state itself, but the paths through time available from the state. We concern ourselves with the possibilities available at each juncture of the mdp.

To this end, for $\pi \in \Pi$, consider the mapping of $\pi \mapsto (\mathbf{I}-\gamma {\mathbf{T}}^{\pi }{)}^{-1}$ (where ${\mathbf{T}}^{\pi }(s,s^{\prime })\stackrel{\mathrm{def}}{=}T(s,\pi (s),s^{\prime })$); in other words, each policy $\pi$ maps to a function mapping each state $s_0$ to a discounted state visitation frequency vector ${\mathbf{f}}_{s_0}^{\pi }$, which we call a possibility. The meaning of each frequency vector is: starting in state $s_0$ and following policy $\pi$, what sequence of states $s_0,s_1,\dots$ do we visit in the future? States visited later in the sequence are discounted according to $\gamma$: the sequence $s_0{s}_1{s}_2{s}_2\dots$ would induce 1 visitation frequency on $s_0$, $\gamma$ visitation frequency on $s_1$, and $\frac{{\gamma }^2}{1-\gamma }$ visitation frequency on $s_2$.

The possibility function $\mathcal{F}(s)$ outputs the possibilities available at a given state $s$:

Put differently, the possibilities available are all of the potential film-strips of how-the-future-goes you can induce from the current state.

Possibility isomorphism

We say two rewardless mdps $M$ and $M^{\prime }$ are isomorphic up to possibilities if they induce the same possibilities. Possibility isomorphism captures the essential aspects of an mdp’s structure, while being invariant to state representation, state labelling, action labelling, and the addition of superfluous actions (actions whose results are duplicated by other actions available at that state). Formally, $M{\simeq }_{\mathcal{F}}{M}^{\prime }$ when there exists a bijection $\varphi :\mathcal{S}\to {\mathcal{S}}^{\prime }$ (letting ${\mathbf{P}}_{\varphi }$ be the corresponding $|\mathcal{S}|$-by-$|{\mathcal{S}}^{\prime }|$ permutation matrix) satisfying ${\mathcal{F}}_M(s)=\left\{{\mathbf{P}}_{\varphi }{\mathbf{f}}^{\prime }|{\mathbf{f}}^{\prime }\in {\mathcal{F}}_{M^{\prime }}(\varphi (s))\right\}$ for all $s\in \mathcal{S}$.

This isomorphism is a natural contender¹ for the canonical (finite) mdp isomorphism:

A theorem I proved

$M$ and $M^{\prime }$ are isomorphic up to possibilities iff their directed graphs are isomorphic (and they have the same discount rate).

Representation equivalence

Suppose I give you the following possibility sets, each containing the possibilities for a different state:

Exercise

What can you figure out about the mdp structure? Hint: each entry in the column corresponds to the visitation frequency of a different state; the first entry is always $s_1$, second $s_2$, and third $s_3$.

You can figure out everything: $⟨\mathcal{S},\mathcal{A},T,\gamma ⟩$, up to possibility isomorphism. Solution.

How? Well, the $L_1$ norm of the possibility vector is always $\frac{1}{1-\gamma }$, so you can deduce $\gamma =.75$ easily. The single possibility state must be isolated, so we can mark that down in our graph. Also, it’s in the third entry.

The other two states correspond to the “1” entries in their possibilities, so we can mark that down. The rest follows straightforwardly.

Theorem

Suppose the rewardless mdp $M$ has possibility function $\mathcal{F}$. Given only $\mathcal{F}$,² $M$ can be reconstructed up to possibility isomorphism.

In mdps, the “AU landscape” is the set of optimal value functions for all reward functions over states in that mdp. If you know the optimal value functions for just $|\mathcal{S}|$ reward functions, you can also reconstruct the rewardless mdp structure.³

From the environment (rewardless mdp), you can deduce the AU landscape (all optimal value functions) and all possibilities. From possibilities, you can deduce the environment and the AU landscape. From the AU landscape, you can deduce the environment (and thereby all possibilities).

Appendix: Opportunity cost

Opportunity cost is when an action you take makes you more able to achieve one goal but less able to achieve another. Even this simple world has opportunity cost:

Going to the green state means you can’t get to the purple state as quickly.

On a deep level, why is the world structured such that this happens? Could you imagine a world without opportunity cost of any kind? The answer, again in the rewardless mdp setting, is simple: “yes, but the world would be trivial: you wouldn’t have any choices.” Using a straightforward formalization of opportunity cost, we have:

Existence of opportunity cost

Opportunity cost exists in an environment iff there is a state with more than one possibility.

Philosophically, opportunity cost exists when you have meaningful choices. When you make a choice, you’re necessarily moving away from some potential future but towards another; since you can’t be in more than one place at the same time, opportunity cost follows. Equivalently, we assumed the agent isn’t infinitely farsighted ($\gamma <1$); if it were, it would be possible to be in “more than one place at the same time”, in a sense (thanks to Rohin Shah for this interpretation).

While understanding opportunity cost may seem like a side-quest, each insight is another brick in the edifice of our understanding of the incentives of goal-directed agency.

Notes

• Just as game theory is a great abstraction for modeling competitive and cooperative dynamics, AU landscape is great for thinking about consequences: it automatically excludes irrelevant details about the world state. We can think about the effects of events without needing a specific utility function or ontology to evaluate them. In multi-agent systems, we can straightforwardly predict the impact the agents have on each other and the world.
• “Objective impact to a location” means that agents whose plans route through the location tend to be objectively impacted.
• The landscape is not the territory: AU is calculated with respect to an agent’s beliefs, not necessarily with respect to what really “could” or will happen.