Can a neural network, seeing only the screen, learn to play Doom well — from nothing but a reward signal?
▶
The agents playing
Two agents, same from-scratch PPO, same pixels-in reward-out setup — but very different jobs. One holds its ground; the other has to move.
The turret — defend the center: rooted in place, it learns to spin and shoot as monsters close in from every side (return ≈ 11).The explorer — deadly corridor: it has to move, fighting its way down a hallway of enemies to reach the armor at the far end.
◇
Inside the build
Each stage of the calculation. Click one to expand its full write-up and validation figure.
01the environment — pixels in, actions out▾
ViZDoom exposes the Doom engine as an RL environment. The agent sees only the framebuffer — grayscaled, downsized to 84×84, with the last 4 frames stacked so a single observation carries motion. Frame-skip repeats each chosen action for 4 tics, which is both faster and easier to learn from. On defend_the_center the reward is +1 per kill and −1 on death.
↳ feeds observations → the policy
02parallel environments — feeding the GPU▾
On-policy RL is data-hungry, so 16 Doom instances run at once. The naive version stepped them one-by-one at ~11 steps/s; because ViZDoom’s engine step is a C++ call that releases the Python GIL, stepping them in a thread pool runs the games concurrently and hits ~115 steps/s — a 10× speedup that turns a 2-day run into a few hours.
↳ feeds rollouts → PPO
03the policy network — a convolutional actor-critic▾
The classic “Nature CNN” — three convolutional layers into a 512-unit dense layer — reads the stacked frames and splits into two heads: a softmax over actions (the policy) and a single value estimate (the critic that judges how good the current state is). About 1.7 M parameters.
↳ feeds action + value → PPO
04PPO — the learning algorithm▾
Proximal Policy Optimization, written from scratch: collect a short rollout, estimate how much better each action was than expected with Generalized Advantage Estimation, then take a few gradient steps on the clipped surrogate objective. The clip keeps each update from moving the policy too far — the trick that makes PPO stable where naive policy gradients thrash. An entropy bonus keeps it exploring.
↳ feeds gradient updates → the policy
05training & recording — watching it learn▾
The same PPO trains two agents on a single GTX 1080. The turret (defend_the_center) goes from dying instantly to holding the center, return climbing ≈ 0 → 11 over 1.5 M steps. The explorer (deadly_corridor) has a richer action set — advance, strafe, turn, shoot — and must fight down a hallway to reach armor; over 3 M steps it cracks the corridor around 500k steps. Because ViZDoom reports the agent’s exact position each frame, its whole trajectory is logged and drawn from above — a top-down “God view” of it reaching deeper into the level as it learns.
◇
Figures
Click any figure to enlarge. Use ← → to move through the set.
◇
What it shows
Yes — with nothing but pixels and a reward signal, PPO learns a genuinely competent Doom policy. The learning curve is the clean, monotonic climb you hope for, plateauing near return 11 after about a million steps.
The engineering matters as much as the algorithm. The single biggest lever here was not a hyperparameter but realizing ViZDoom releases the GIL, so the environments could be stepped in parallel threads for a 10× throughput win — the difference between a feasible run and a two-day one.
The same agent, two very different skills. A turret that learns to hold ground and an explorer that learns to navigate and fight down a corridor come out of the identical PPO — the difference is only the action set and reward. And because ViZDoom exposes the agent’s position, the explorer’s learning can be watched in space: early paths die at the spawn, later ones push all the way to the exit.
Written from scratch — the CNN actor-critic, GAE, and the clipped PPO objective are all in ~200 lines, no RL library — so the whole pipeline is legible end to end: pixels in, a Doom-playing policy out.
▶
Now you play
The same game, for you — the original Doom (shareware episode 1, “Knee-Deep in the Dead”), running in your browser through a self-hosted DOS emulator. Nothing is installed or streamed from elsewhere; it opens in a new tab so it has the whole screen.