How Deep Reinforcement Learning Learned to Play Atari

How does a computer learn to keep a ball in play without someone writing a rule for every bounce? Deep reinforcement learning combines a neural network with learning from actions and rewards. In a landmark 2015 Atari study, a deep Q-network, or DQN, used visual input to learn which actions were likely to lead to higher future rewards.

The achievement was a versatile learning method tested across 49 games. It was not one player learning all the games at once, and it was not proof of human-like understanding. Mnih and colleagues, Human-level control through deep reinforcement learning.

How learning from rewards works

Think of a game in which a paddle has to keep a ball in play. The program receives a view of what is happening, chooses an action and receives feedback through rewards.

An action can be useful even if it earns no points immediately. Moving the paddle into position may set up a successful return a moment later. The learning problem is to work out which choices tend to lead to better outcomes over time.

DQN estimates the future reward associated with available actions. During training, it also explores, sometimes choosing an action rather than always following its current best estimate. This gives it opportunities to learn from outcomes it would otherwise miss. Mnih and colleagues, algorithm and methods.

What the neural network contributes

Game screens contain a lot of visual information. Earlier approaches often depended on people defining useful features for the learning system.

DQN used a convolutional neural network to learn useful representations from processed game frames. Its output estimated the value of each available action. Learning therefore connected what appeared on the screen with what the agent should do next.

The same algorithm, network design and training settings were applied across the games. Each game still had its own training. Reusing a method across tasks is different from transferring everything learned in one game into another. Mnih and colleagues, experimental design.

Why the system revisited old experience

Learning only from the most recent sequence of play can be unstable. Consecutive frames often resemble one another, and changing the network changes the decisions that generate its next experiences.

The researchers used experience replay: storing transitions from play and sampling them later for training. A stored transition includes what the agent observed, what it did, the reward and what happened next.

They also used a separate target network that was updated periodically. This made the values used as learning targets more stable between updates. The paper tested the importance of these components by examining performance when they were removed. Mnih and colleagues, learning method and component comparisons.

What human level meant in the study

The researchers compared game scores with those of a professional human games tester under controlled conditions. DQN reached more than 75 percent of the human score on 29 of the 49 games.

That is a substantial result, but it is more precise than saying it beat humans at every game. Tasks requiring longer-term planning remained difficult. The paper identifies Montezuma’s Revenge as one such challenge. Mnih and colleagues, results and discussion.

Performance also followed extensive training. The reported training schedule used 50 million game frames. A strong final score does not tell us that the program learned as quickly as a person.

What this tells us about brains

The researchers drew inspiration from neuroscience, including ideas about reward learning and replaying experience. That connection helped motivate the design.

But the experiment measured a computer system playing games. It did not test a therapy, measure human neuroplasticity or establish that people learn in exactly the same way. Its relevance to brain research is as a model and source of questions, not a direct instruction for changing behaviour or treating pain.

For a general reader, the paper illustrates an important distinction: a system can learn an effective action strategy from experience without demonstrating everything we mean by human understanding. Its achievement is clearest when we name the task, the comparison and the limits.

Read an AI result with these questions in mind

When you encounter a claim of human-level AI, look for the task, the human comparison and the training required. Then ask what happens outside that test. This is a useful way to read the Atari result and to assess more consequential claims about health AI.