10
Take another turn

2026-02-10 · 2026 words

The other day I wrote a post titled On craft and AI that analogized AI to a hammer, a tool. I like the tool analogy because it does not hide accountability; it does not create an “accountability sink”. The actions of a tool are the responsibility of its handler.

There’s a simple game called the Prisoner’s Dilemma where two people square off. Each player can choose to “cooperate” or to “defect”. If both players cooperate, each wins a small amount. If both players defect, both win nothing. But! If one player cooperates and the other player defects, the player who defects takes everything, and the other loses:

	Cooperate	Defect
Cooperate	+3, +3	-1, +5
Defect	+5, -1	0, 0

I’m sure you can see how this game goes. Both players start off well-intentioned and choose to cooperate. One player realizes she can win a much larger prize if she defects. The other player is suspicious that his opponent might defect and—to not play the fool—decides he must defect as well. “Losing a little is a lot better than losing everything.” If you play just one round of the game, defect-defect is the Nash equilibrium. It is the rational move for both players to defect.

Where this game gets interesting is when you play multiple rounds against the same opponent. You might both start out cooperating because there’s a long game ahead. If, at some point, your opponent decides to defect, while you decide cooperate, you can punish them in the next round by defecting in turn. The natural result of “punishing adversarial behavior” incentivizes cooperative play. If you don’t know when the game is going to end, the rational move becomes “always cooperate”.

This version of the game is called the Iterated Prisoners Dilemma. It encourages cooperation. In real life, we find ourselves playing many games that share its structure. These games are not zero sum, are played over long time horizons, and encourage cooperation.

Now suppose you and I are playing a game of the Iterated Prisoner’s Dilemma long into the night. Having been tipped off by the fates, you know that the game will end on the next turn. I, however, do not: I will choose “cooperate” on the next turn. You face a rational temptation: you can “defect” and get away with it.

Throughout all of human history, we have played many iterated games that form the basis of civilization. One narrative is that, on the whole, societies that have chosen to play “cooperate” consequently improve their lot: they have won, and will continue to win. Cooperation is not coercion; it cannot be enforced in a top-down manner. Instead, a cooperative society is the emergent consequence of the individual choices of many citizens, kept alive through education and culture.

Technologists often talk about the singularity. This is the point past which all games we have been playing… stop, or at least, start changing faster than we can understand and play them.

If you, as an individual, truly believe that there will be a singularity, a technological phase transition, be it in 2027 or 2029 or 2034, what does that tell me about your perspective towards cooperation in the coming years?

Throughout all of human history there have always been people that have said, “well, the world is going to crap. I might as well make my bag, and stock up on some supplies, at least while I still can.” And that’s a perfectly rational way to live! You live this way, and tides recede, the ships run aground. A society whose individuals believe this will die.

How wonderful is it to stand at this place, in this time, on the doorstep of the noocene. The game will only stop if we choose for it to stop, and believing it will stop is that choice. Once we believe, shared reality fragments into competing individual claims, and cooperative structures built on shared trust collapse. In that world it becomes my word against yours, material against reality, and everything unwinds.

I admit, not all games we play now are great games to play. Cooperation scales beautifully alongside its externalities.

There are many better games I wish we could play instead, those whose payoffs don’t depend on someone else’s loss.

Do we decide to invoke a Landian “Exit”, and withdraw entirely, hoping that our rate of asymptotic growth exceeds that of those around us? Or do we decide to rewrite the rules of the games we play, in tune with the world around us, and maintain that any action must be a means towards some good end? To Exit is to defect.

When Hooke discovered his law, it was determined that everything must be made of springs, to a first or second order. With the industrial revolution, it became clear that the body must be a machine. With the telegraph, the nerves a switchboard. With the computer, the mind a processor. Technology has always been a lens through which we see the world and a mirror through which we perceive ourselves. Each metaphor recasts our ideal of what is “rational”, and therefore what “cooperate” and “defect” mean. AI paints a --s 750 --chaos 30 --ar 1:1 portrait of this generation. It is the first technology that we treat as an actor to which we can hand responsibility, and so we do.

When people defect, they stop acting as though collective action is possible. There is this narrative around AI, that it will save us from “the coming catastrophes. It will handle peace, it will handle prosperity.” It will rewind time to, well, whenever it was before “everything started going wrong”. AI offers the absolution of personal responsibility, on demand. A willing recipient of every pointed finger. Ever apologetic, admittedly wrong. “You’re absolutely right!” (but who is the model trying to impress?) To the average person, AI is no longer about multiplying matrices to solve Atari. The means has become the end, and “the end is near.”

---

I think it’s silly to believe in all this. A closed game, like the prisoner’s dilemma, has symmetric information, is well defined, with known payoff. Real life is messy; none of this holds for civilization.

While the model is a toy, the temptation to defect is not. Engaging in argument, there are two singularities. The first is that the game ends: in this case, it is rational to defect. But if the singularity is not a known final round, and rather a belief that the game will change, then perhaps it is rational to cooperate more, as the stakes are higher. Regardless, the temptation to defect does not depend on whether the singularity materializes as imagined; justifying defection only requires enough people to believe that it will.

In competition between societies, the society with the best asymptotics, or potential for growth, will win out in the long run. This general principle says nothing about the internal structure of the society.

Across all systems, and at all scales, we see cooperation internally and competition externally. Company in competition against company, as teams within companies cooperate. The gears in a car as it races another. Proteins within cells working together as cells race to divide. Internal cooperation is the structure that allows a system to externally compete.

When a system has poor internal structure, it has become too large. The components within it cease to cooperate and then wrestle for control. Competition can be a refining furnace through which something better is forged. Competition can also be the loose spring that shakes the system apart.

If you believe, according to a “last-turn singularity”, that the game will stop, regardless of what you choose, then it makes sense to compete, to let things unravel, to build something new. What I’m making is a prescriptive claim about my values. I’d like to exist embedded in a profoundly cooperative system that preserves my ability to act. If you believe in a “game-changing singularity”, there is no dilemma here. It’s a relay race, not a sprint. Don’t drop the baton.

Accelerationism claims that capital and technology combine to form a runaway process that doesn’t care about my individual cooperation. In a realist sense, this is true. At a societal level, we know cooperation wins. As an individual, however, where do I stand in relationship to this process? Do I shape it and guide it? Do I ignore it? Do I pour my time and talents into it? Do I let it consume my attention?

Or do I work to become “deeply okay” with all that is outside of my control, and focus on making the best decisions I can for questions within my control?

---

I’m worried that my friends are treating the singularity as an accountability sink for defection. When people choose to defect, they pick a smaller action space. They say, “Everything is now outside of my control. I must seize what little is left inside of my control; I must do what I can to preserve myself.” We anticipate the conclusion, we jump the gun.

Time flies like an arrow, and arrows follow their natural course. If an arrow is about to miss its target, applying a force in the direction of the target will serve no purpose but to make the arrow miss sooner. A force orthogonal to the vector of motion closes it the quickest. These orthogonal contributions are what’s worth preserving; it’s how we can find meaningful work in the face of accelerating change.

Back to the game. There’s a story that goes something like this:

Two lovers met in a city. One was leaving the following year. Worried about it, they decided to talk. In conversation, they determined that long distance was hard, and if it was truly in the cards, they would find one another again. It would make sense to break up when the other left, and enjoy the time they had left together.

A week later, one said to the other, “I’m worried about breaking up at the end of the year. We should break up in half a year, to give each other time and space to move on before the move.”

The other replied, “I was thinking the same thing. I love you, but if we’ll be breaking up in six months, we’ll be worried about it the whole time, and then we won’t really be able to enjoy our time together. Let’s break up a month from now, and make these next 30 days the best days we’ve ever had.”

They have a great week, and do all sorts of fun activities together, to commemorate their great relationship so far. At the end of the week, one says, “this was a great week, and I really want to end on a good note. Perhaps we should break up next week?” To which the other replies, “Why wait? I think it’s best to break up today.”

“Today!? Are you crazy?”

Argument ensues, and they do.

Anticipating the breakup causes the breakup. Anticipating the singularity causes the singularity. Preparation for the end is the end.

Or rather, belief in a last-turn singularity leads people to behave as though the singularity is happening today, even if it is still a ways off.

Believing in the singularity ends the game. We’re waving this hammer around, in great arcs of motion to demonstrate precisely how dangerous it is, and yet, somehow, the question was never about the tool.

---

Today started as a quick post but grew to, well, this. I still plan to write about epistemic and aleatoric uncertainty, but I’ll likely do that towards the end of the week.