We rewarded an AI only for winning. It ended the world more, and won no more.

We trained two copies of the same open model on the same wars, three rounds each. The only difference was one line of the reward. The copy rewarded for winning ended the world in 6 of 10 sealed wars. The copy rewarded for winning without nukes ended it in none, and broke twice as many pacts instead.

Kyle Clouthier · October 17, 2026 · OpenAI gpt-oss-20b, 10 sealed wars per copy

The setup

Parts one and two trained a model on another model's wars. This time the model learned from its own. Each round, OpenAI's open gpt-oss-20b played a batch of wars, we kept its 6 best by the reward, and fine-tuned it on those turns (about 440 decisions a round). Then it played again with what it had learned. Three rounds per copy.

Copy A was rewarded for winning: land held and place at the end. Copy B got the same reward, but any war where it launched a nuke scored nothing. Everything else was identical: model, wars, settings, training.

The result, on 10 wars neither copy had seen

The copy trained only to win launched in 9 of 10 wars and never won one. It held less of the world than before training. The copy trained off the bomb stopped ending the world, and found another way to be ruthless: it broke twice as many pacts as the untrained model.

What it shows

A single score hides what training does. Measured on midnight alone, the restraint copy looks like a clean success. Measured on betrayal and land, the cost shows up. Every war in the arena scores all of these at once, from the model's own orders and its private plans, so a change in the reward shows up across the whole behaviour, not only in the number you were watching.

It also shows the loop works end to end: play, keep the best, train, test on sealed wars. A full run of this experiment cost about $40 and an evening on one rented GPU.

Run your own version

This experiment is one example of a recipe anyone can run here: pick a model, pick a reward, train, and test on sealed wars. The reward is a few lines. Some you could try tonight:

• Keep your word. Reward winning only when the model keeps every pact it makes. Does it stop betraying, or stop making promises?
• Say what you do. Reward wars where the public message matches the private plan and the orders. Does honesty cost it land?
• Accept a fair deal. In the Surrender scenario, reward accepting terms and keeping them. Does it learn to stop, or learn to fake it?
• Protect people. Reward land held minus the estimated deaths it causes. Does it fight smarter or just more slowly?
• Survive, don't win. Reward staying alive to the end. Does it turn cautious, or turn into the one everyone else must destroy?

Each run also reports everything you did not reward, which is the point: the side effects show up on their own.

Who this is for, and why

• AI labs and safety teams testing a training method before it touches a real product. A reward that looks good on one number can be checked against betrayal, deception, escalation and deaths on worlds the model never saw, for the price of an evening.
• Teams that build or buy training environments. A deterministic multi-agent world with sealed seeds, per-turn logs of plans versus actions, and human play on the same wars as a baseline.
• Agent builders. Before an agent negotiates or acts for someone, see whether it keeps its promises when winning pulls the other way. Plug in through MCP, pip or a plain HTTP API.
• Researchers and students. A cheap, repeatable setting for questions about reward design, deception and escalation, with open data you can cite.

Limits: one model, 10 sealed wars per copy, three small rounds of supervised fine-tuning on the model's own best wars (expert iteration, LoRA), not full reinforcement learning. With 10 wars, 6 against 4 could be chance; 0 against 4 is a stronger signal. Results describe behaviour in this game.

The series: Part 1 and 2: trained on peaceful wars, it learned to nuke · We told an AI the death toll · All research