Module 2: Machine Learning Fundamentals

5. Reinforcement Learning: Learning by Trial and Error

Next →

Description

In this lesson you'll understand Reinforcement Learning: the type of Machine Learning where an agent learns to make decisions in an environment by receiving rewards or penalties based on its actions. Unlike supervised (where you give it correct answers) or unsupervised (where you look for patterns without labels), here the agent learns by trial and error: it tries actions, receives feedback (a high or low reward), and adjusts its behavior to maximize rewards over the long term.

This matters because some of the most impressive advances in AI (AlphaGo, robots that learn to walk, agents in video games) use reinforcement learning. Although as an AI Engineer you won't work with RL as often as with supervised (most APIs and LLMs are supervised or generative), understanding the concept helps you follow conversations about agents, read news about DeepMind or OpenAI, and reason about systems that "learn from experience" instead of from labeled data.

What is Reinforcement Learning?

Definition: A type of Machine Learning where an agent learns to take actions in an environment in order to maximize a cumulative reward over the long term. There's no data labeled with "correct answers"; the agent learns by trial and error, receiving feedback (rewards or penalties) after each action.

Key components:

  • Agent: The system that makes decisions. E.g. a chess player, a robot, a character in a video game.
  • Environment: The world in which the agent acts. E.g. a chessboard, a physical space, a video game.
  • Actions: The options the agent can choose at each moment. E.g. move a piece, turn left, jump.
  • State: The current situation of the environment. E.g. the position of the pieces on the board, the robot's location.
  • Reward: Numeric feedback the agent receives after an action. E.g. +10 if it wins the game, -1 if it loses, 0 if nothing has happened yet.

The agent's goal: To learn a policy (strategy): which action to take in each state in order to maximize the cumulative reward over the long term.

Analogy: It's like teaching a dog tricks: you don't give it step-by-step instructions (that would be supervised); you give it a reward (a treat) when it does something right and nothing (or a penalty) when it does something wrong. The dog tries actions at random at first; over time, it learns which actions lead to rewards (e.g. "sit" → treat) and repeats those actions. That's reinforcement learning: learning by trial and error with feedback.

Basic Reinforcement Learning flow

Agent observes state → Chooses action → Environment gives reward + new state
Agent adjusts policy → Repeats (thousands of episodes)
Result: A policy that maximizes cumulative reward

Concrete example (learning to play Go):

  1. Agent: The system that plays Go (e.g. AlphaGo).
  2. Environment: The Go board (state = the position of the stones).
  3. Actions: Placing a stone at a position on the board.
  4. Reward: +1 if it wins the game, -1 if it loses, 0 during play.
  5. Learning: The agent plays thousands (or millions) of games against itself or others. At first it plays badly (nearly random actions); over time, it learns which moves lead to wins (a high reward) and avoids moves that lead to losses.
  6. Result: A policy that chooses good moves in each state of the game.

Key point: You didn't give the agent a list of "correct moves" (that would be supervised). You gave it a goal (to win) and the agent learned on its own which actions lead to that goal.

Differences from Supervised and Unsupervised

AspectSupervisedUnsupervisedReinforcement
DataExamples with labels (correct answers)Data without labelsSequences of (state, action, reward)
GoalPredicting labels on new dataFinding patterns or structureMaximizing cumulative reward over the long term
FeedbackThe correct answer for each exampleNo feedbackA reward (positive or negative) per action
LearningMinimizing the difference between prediction and labelFinding similarities or reducing dimensionsAdjusting the policy to maximize reward
ExampleSpam filter (classifying emails)Customer clusteringAlphaGo (playing Go)

Note: In supervised, the feedback is immediate and explicit ("this is the correct answer"). In reinforcement, the feedback is a numeric reward that can arrive with a delay (e.g. you win the game at the end, but you made decisions throughout the whole game; you have to learn which decisions contributed to winning).

Examples of Reinforcement Learning

1. AlphaGo (DeepMind, 2016)

  • Agent: A system that plays Go.
  • Environment: The Go board (19x19).
  • Actions: Placing a stone at a position.
  • Reward: +1 if it wins, -1 if it loses.
  • Learning: It played millions of games against itself; it learned strategies that let it beat the human world champion.

Impact: It showed that RL can master complex games with an enormous decision space.

2. Robots that learn to walk

  • Agent: A humanoid or quadruped robot.
  • Environment: Physical space (or a simulation).
  • Actions: Moving joints (knee, ankle angles, etc.).
  • Reward: +1 for each step forward without falling, -10 if it falls.
  • Learning: It tries random movements at first; over time, it learns which movements lead to walking without falling.

Note: Training robots with RL is usually done in simulation (faster and safer) and then the learned policy is transferred to the real robot.

3. Video games (Atari, Dota, StarCraft)

  • Agent: A system that plays the video game.
  • Environment: The game (screen, score, etc.).
  • Actions: Controller buttons (up, down, shoot, etc.).
  • Reward: Game points, winning/losing.
  • Learning: It plays thousands of games; it learns strategies that maximize points or the probability of winning.

Famous examples: DQN (Atari), OpenAI Five (Dota 2), AlphaStar (StarCraft II).

4. Recommendation systems with feedback

  • Agent: A recommendation system (e.g. YouTube, TikTok).
  • Environment: The user (what they watch, what they do).
  • Actions: Recommending video A, B or C.
  • Reward: +1 if the user watches the video all the way through (engagement), 0 if they ignore it, -1 if they mark "not interested".
  • Learning: It learns which recommendations maximize engagement over the long term (not just immediate clicks, but the user continuing to use the app).

Note: Many modern recommendation systems combine supervised (predicting what you'll like based on your history) and reinforcement (optimizing long-term engagement by adjusting recommendations based on feedback).

Why this matters for an AI Engineer

As an AI Engineer, you won't work with RL as often as with supervised learning (most APIs are supervised or generative models). But understanding RL helps you:

  1. Understand agents: When the AI ecosystem talks about "agents" (systems that plan, use tools, execute steps), they often have RL components (learning which tools to use based on feedback). Understanding RL helps you follow those conversations.

  2. RLHF (Reinforcement Learning from Human Feedback): Modern LLMs (GPT-4, Claude) use RLHF as part of their training: the model generates responses, humans rate them (a reward), and the model adjusts its behavior to maximize rewards. That's RL applied to language models.

  3. Optimizing systems for the long term: If you build a system that must optimize engagement, conversion or user satisfaction over the long term (not just maximize immediate clicks), RL can be relevant. Even if you don't implement it yourself, understanding the concept helps you design the system or evaluate solutions.

  4. Reading papers and news: Many advances in AI (AlphaGo, AlphaStar, robots, agents) use RL. Understanding the concept lets you read those papers or news without getting lost in the terminology.

Limitations and challenges of Reinforcement Learning

1. Exploration vs exploitation

Problem: The agent must balance exploring (trying new actions it doesn't know are good) and exploiting (using actions it already knows give a good reward). If it only explores, it never uses what it learned; if it only exploits, it can get stuck in a suboptimal strategy.

Example: In a game, if the agent only exploits (repeating the strategy it already knows), it may not discover better strategies. If it only explores (trying random actions), it never takes advantage of what it learned.

2. Delayed reward

Problem: In many problems, the reward arrives at the end (e.g. you win or lose the game after many decisions). The agent has to learn which decisions contributed to the final reward; that's hard (the credit assignment problem).

Example: In chess, if you win the game, which moves were good and which were bad? The last move won, but earlier moves also contributed (or hurt). The agent has to learn that relationship.

3. Computational cost

Problem: RL usually requires millions of episodes (games, attempts, simulations) in order to learn. That's expensive in time and compute, especially if the environment is physical (a real robot) or complex (a large video game).

Solution: Use simulation (faster and safer) and then transfer the learned policy to the real environment. Or use more efficient techniques (e.g. model-based RL).

4. Reward design

Problem: Defining the reward function is critical. If the reward is badly designed, the agent learns undesired behaviors (e.g. it exploits bugs in the environment to maximize reward without doing the useful task).

Example: A cleaning agent that receives a reward for "picking up trash". If the reward is only for "picking up", the agent can throw trash on the floor and then pick it up to maximize reward. Designing rewards that incentivize the correct behavior is an art.

Exercises

Exercise 1: Identify RL components

For the AlphaGo case, identify:

  1. Agent
  2. Environment
  3. Actions
  4. State
  5. Reward
See solution
  1. Agent: The system that plays Go (AlphaGo).
  2. Environment: The Go board (19x19).
  3. Actions: Placing a stone at a position on the board.
  4. State: The current position of all the stones on the board.
  5. Reward: +1 if it wins the game, -1 if it loses, 0 during play.

Exercise 2: Is it RL?

Classify each problem as supervised, unsupervised or reinforcement learning. Justify in one sentence.

  1. Training a robot to walk in a physical space (it receives a reward for each step forward without falling).
  2. Classifying emails as spam or not-spam (with labeled emails).
  3. Grouping customers into segments (without predefined labels).
  4. Training an agent to play chess (it receives a reward if it wins).
  5. Predicting the price of a house from its features (with known training prices).
See solution
  1. Reinforcement Learning. The robot learns by trial and error, receiving rewards (there are no labels for "the correct way to walk").
  2. Supervised Learning (classification). There are emails labeled spam/not-spam.
  3. Unsupervised Learning (clustering). There are no labels; the model groups by similarities.
  4. Reinforcement Learning. The agent learns by trial and error (it plays games, receives a reward if it wins).
  5. Supervised Learning (regression). There are known prices (numeric labels).

Exercise 3: RLHF (Reinforcement Learning from Human Feedback)

Modern LLMs (GPT-4, Claude) use RLHF as part of their training: the model generates responses, humans rate them (a reward), and the model adjusts its behavior to maximize rewards. Why is this reinforcement learning and not supervised learning? Write 2-3 sentences.

See answer guide

Possible guide: In supervised learning, you'd give the model pairs of (question, correct answer) and the model would learn to predict that answer. In RLHF, the model generates responses (actions), humans rate them with a score (a reward), and the model learns to generate responses that receive high scores. There's no single predefined "correct answer"; there are many possible responses and the model learns which ones are better according to human feedback. That's reinforcement learning: learning to maximize a reward (a human score) through trial and adjustment.

Exercise 4: Limitations

What's the main limitation of reinforcement learning? Write 2-3 sentences.

See answer guide

Possible guide: The main limitation is computational cost: the agent needs to try millions of episodes (games, attempts, simulations) in order to learn, and that's expensive in time and compute. Besides, designing the reward function is critical; if it's badly designed, the agent can learn undesired behaviors (exploiting bugs in the environment to maximize reward without doing the useful task). Finally, the reward usually arrives with a delay (e.g. you win the game at the end), and the agent has to learn which earlier decisions contributed to that reward (the credit assignment problem).

Exercise 5: Why it matters for AI Engineering

Why does an AI Engineer need to understand reinforcement learning, even though most APIs are supervised models? Write 2 reasons.

See answer guide

Possible guide:

  1. Understanding agents and RLHF: Modern "agents" (systems that plan, use tools, execute steps) often have RL components (they learn which tools to use based on feedback). Modern LLMs (GPT-4, Claude) use RLHF to improve response quality. Understanding RL helps you follow those conversations and read technical documentation.
  2. Reading papers and news: Many advances in AI (AlphaGo, AlphaStar, robots) use RL. When you read papers from DeepMind or OpenAI, understanding basic RL concepts lets you follow the thread without getting lost in terminology.

Summary

In one sentence: Reinforcement Learning is the type of ML where an agent learns by trial and error in an environment, receiving rewards or penalties based on its actions, with the goal of maximizing cumulative reward over the long term.

Key points:

  • Components: Agent, environment, actions, state, reward, policy (strategy).
  • Flow: The agent observes a state → Chooses an action → Receives a reward + a new state → Adjusts its policy → Repeats (millions of times).
  • Difference from supervised: There are no predefined correct answers; the agent learns from feedback (a reward).
  • Difference from unsupervised: It doesn't look for patterns in static data; it learns to make decisions in a dynamic environment.
  • Famous examples: AlphaGo, robots that learn to walk, agents in video games (Atari, Dota, StarCraft).
  • Relevance for an AI Engineer: Understanding agents, RLHF (used in modern LLMs), reading papers and news about advances in AI.
  • Limitations: Computational cost (millions of episodes), exploration vs exploitation, delayed reward, reward function design.

Connection with the rest of the module and the guide

In the next lesson you'll see Training vs Inference, the most important distinction for AI Engineering. That distinction applies to supervised, unsupervised and reinforcement: in every case there's a learning phase (training: the model or agent learns from data, patterns or rewards) and a usage phase (inference: the already-trained model makes predictions or takes decisions).

In later modules:

  • Module 5 (LLMs): You'll see that GPT-4, Claude and others use RLHF (Reinforcement Learning from Human Feedback) as part of their training. Understanding RL helps you follow that explanation.
  • Module 7 (AI Engineering): You'll know that your role (integrating models, not training them) applies to supervised models, generative models and sometimes agents with RL components.

Additional resources

  1. DeepMind: AlphaGo Documentary — A documentary about AlphaGo and how it beat the world Go champion. In English. Very visual and inspiring.

  2. OpenAI: Spinning Up in Deep RL — An introduction to RL with examples and code. In English. More technical; for after this guide if you want to program agents.

  3. Coursera: Reinforcement Learning Specialization — A complete RL course. In English. More mathematical; for going deeper after this guide.

  4. Sutton & Barto: Reinforcement Learning Book — The reference book on RL (free online). In English. Very technical; for after this guide if you want academic depth.

  5. OpenAI: Learning from Human Feedback — An article about RLHF, used in GPT-4 and other LLMs. In English. Useful for understanding the connection with Module 5 (LLMs).

  6. Google AI: Reinforcement Learning Explained — A brief, visual introduction. In English.

← Previous4. Unsupervised Learning: Finding Patterns without LabelsNext →6. Training vs Inference: The Most Important Distinction for AI Engineering