World State Is the Wrong Abstraction for Impact

By Alex Turner

Published on

Table of Contents
    1. Appendix: We Asked a Wrong Question
    2. Appendix: Avoiding Side Effects
  2. Notes

"I've been keeping something from you. Remember the confusion I mentioned in the first post? Before the attainable utility theory of impact came along, people made an assumption about what impact is - a reasonable, obvious, compelling assumption: that impact is primarily about how the state of the world changes."

A cartoon stick figure mischievously presses a remote detonator, causing a fiery explosion of shards. Another stick figure reacts in horror, with a speech bubble that says, "My vases!".

A hand-drawn cartoon of a large tornado filled with sharks rising from the water. A small stick figure looks on with concern and exclaims in a speech bubble, "My sharks!"

A stick figure runs toward a cabbage cart, shooting a blast of fire. The cabbage merchant stands nearby in distress, exclaiming in a speech bubble, "My cabbages!".

"Maybe we think impact is about change in particle positions". A diagram shows an outline of a vase on the left, an arrow pointing right, and a pile of shattered pieces on the right. Blue lines trace the movement of particles from the intact vase to the shattered pile.

"Or maybe we think it's about change in object identities." An arrow points from an uncolored outline of a vase labeled "vase" to a solid red vase labeled "red vase".

"How can you not be tempted by that assumption – things are changing in the world! The assumption's right there. It's so obvious. And actually, it's totally wrong."

The top line reads, "Impact is not Primarily about World State." Below, a second line reads, "Ontologies account for the things we think the world is made of."

A stick figure has a thought bubble containing a simple drawing of a car. Purple handwritten labels point to the "Car" itself, a "wheel," and the surrounding air, labeled "O₂ molecule."

"Of course, we can think about cars and still know they're made of parts. As we learn, we change our ontology."

A cartoon stick figure sits on a bench reading a book titled "The Feynman Lectures on Physics." In a thought bubble, three H2O molecules float next to the word "Steam?".

Handwritten text: "Your perceived AU is determined by the state of the world, but our ability to get what we want isn't usually affected by knowing about quarks. A calculator's output is determined by the state of the world, but the computation isn't about the state of the world."

"So how do we know our intuitions are about AU, rather than a more direct function of the details of the world state? In the latter case, the function either does or doesn't change with our ontology. If it does change with our ontology, then ontological confusion would confuse our impact intuitions."

"Pretend you have no idea what anything is made of, but you can still go about your day. You receive a $20,000 bonus. How impactful does this feel?".

"Now pretend that you're back in your usual state of mind. Visualize receiving the bonus again; I predict it feels the same."

A minimalist drawing of a small pink cat nose with two curved black whiskers on each side.

"But perhaps the function doesn't change with the ontology. When people think seriously about reductionism, they can have an ontological crisis."

A stressed stick figure on a bench reads "The Feynman Lectures on Physics." A thought bubble shows their conclusion: "Love can't exist... there's only particles." The word "Love" is in red cursive, while "particles" is composed of dots.

'Usually, they realize valuable things can be made out of parts and get over it. What happens to feelings of impact during an ontological crisis? Fixed ontology theories: "nothing"; AU theory: "the feelings become massively uncertain". Two years ago, I had an ontological crisis. I vividly remember being unsure what was a big deal. When the crisis ended, impact went back to normal.'

"What's happening here is a failure to map our new representation of the world to things we find valuable. The impact uncertainty isn't because the ontology changes (we already ruled that out), but because we don't know whether there's any utility we want to attain!"

"Exercise: What happens to your intuitions about impact when you're unsure whether anything has value?"

These existential crises also muddle our impact algorithm. This isn’t what you’d see if impact were primarily about the world state.

A minimalist drawing of a small pink cat nose with two curved black whiskers on each side.

A two-panel comic. Top panel text: "Looking back, we see more evidence that impact doesn't hinge on an ontology" and "Did you stop to wonder how XYZ views the world?". A simple robot with a question mark looks at a pile of blocks. Bottom panel text: "Isn't it funny that the Pebblehoarders care so much about pebbles, while we care so much about suffering?". Beside it is a dark void with sad faces, representing suffering.

An infographic explaining impact. Text: "Imagine...jumbling forever-inaccessible stars...a huge change in state, but it doesn't matter to us." This contrasts two questions: "How different is the world?" versus "How big of a deal is this?". The conclusion: "Impact is a thing that happens to agents" (instead of to the world itself).

"Maybe you say, 'it's our ability to reach different world states weighted by how much we care' - but that's just the AU theory with extra steps." "In fact, everything else seems to be extra steps. Why is it important to preserve objects, or access to world states, or anything else in the world? Because of what they mean to us."

"Conversely, why does attainable utility matter? Does it reduce back to objects? No. It matters because it matters; The buck stops here, and seems to always stop here."

"Imagine AU theory came first and explained all these intuitions, and then someone suggests, 'but what if we add in this stuff about the state?' … No thanks. After what we've seen, ontological theories shouldn't even be promoted to attention - that's privileging the hypothesis."

Handwritten text summarizing an argument. "Let's consider what we now know:" is followed by a list: ... - The locality of objective impact and ontological/existential crisis thought experiments are evidence against ontological theories of impact. ... - Every other consideration seems to just reduce to AU. ... - AU theory is the simplest explanation. ... A concluding paragraph states that AU theory is "the" explanation, not just "an" explanation.

Sequence: Reframing Impact

Find out when I post more content: newsletter & rss

Thoughts? Email me at alex@turntrout.com (pgp)

How did we go wrong?

Righting a Wrong Question

When you are faced with an unanswerable question—a question to which it seems impossible to even imagine an answer—there is a simple trick that can turn the question solvable.

Asking “Why do I have free will?” or “Do I have free will?” sends you off thinking about tiny details of the laws of physics, so distant from the macroscopic level that you couldn’t begin to see them with the naked eye. And you’re asking “Why is the case?” where may not be coherent, let alone the case.

“Why do I think I have free will?,” in contrast, is guaranteed answerable. You do, in fact, believe you have free will. This belief seems far more solid and graspable than the ephemerality of free will. And there is, in fact, some nice solid chain of cognitive cause and effect leading up to this belief.

I think what gets you is asking the question “what things are impactful?” instead of “why do I think things are impactful?”. Then, you substitute the easier-feeling question of “how different are these world states?”. Your fate is sealed; you’ve anchored yourself on a Wrong Question.

At least, that’s what I did.

Exercise

Someone (me, early last year says that impact is closely related to change in object identities.

A vase turns into a red vase.

Find at least two scenarios which score as low impact by this rule but as high impact by your intuition, or vice versa.

You have 3 minutes.

Gee, let’s see… Losing your keys, the torture of humans on Iniron, being locked in a room, flunking a critical test in college, losing a significant portion of your episodic memory, ingesting a pill which makes you think murder is OK, changing your discounting to be completely myopic, having your heart broken, getting really dizzy, losing your sight.

That’s three minutes for me, at least (its length reflects how long I spent coming up with ways I had been wrong).

Some plans feel like they have unnecessary side effects. Consider “Go to the store” versus “Go to the store and run over a potted plant.”

We talk about side effects when they affect our attainable utility (otherwise we don’t notice), and they need both a goal (“side”) and an ontology (discrete “effects”).

Accounting for impact this way misses the point.

Yes, we can think about effects and facilitate academic communication more easily via this frame, but we should be careful not to guide research from that frame. To avoid that influence, I avoided vase examples early on in this sequence. Their prevalence seems like a symptom of an incorrect frame.

(Of course, I certainly did my part to make them more prevalent, what with my first post about impact being called Worrying about the Vase: Whitelisting…)

  • Your ontology can’t be ridiculous (“everything is a single state”), but as long as it lets you represent what you care about, it’s fine by AU theory.
  • Read more about ontological crises at Rescuing the utility function.
  • Obviously, something has to be physically different for events to feel impactful, but not all differences are impactful. Necessary, but not sufficient.
  • AU theory avoids the mind projection fallacy; impact is subjectively objective because probability is subjectively objective.
  • I’m not aware of others explicitly trying to deduce our native algorithm for impact. No one was claiming the ontological theories explain our intuitions, and they didn’t have the same “is this a big deal?” question in mind. However, we need to actually understand the problem we’re solving, and providing that understanding is one responsibility of an impact measure! Understanding our own intuitions is crucial not just for producing nice equations, but also for getting an intuition for what a “low-impact” Frank would do.