Month 5 · ~17 h of video + practice

Teaching robots instead of programming them

Your goal this month: learn imitation learning and reinforcement learning, the two ways robots learn skills from data.

This is the month everything connects.

Reinforcement learning is the dynamic programming from Month 4, done without knowing the rules of the world in advance.

month 50/9 · 0%

Watch

Marco Pavone and Daniele Gammelli, Stanford

The same course as Month 4, now moving from control to learning.

Lecture 13: intro to learning
Lecture 14: intro to imitation learning and RL
Lecture 15: imitation learning
Lecture 16: fundamentals of RL
Lecture 17: value-based methods like Q-learning
Lecture 18: policy optimization
Lecture 19: model-based RL, and the whole stack from perception down to the motors

Chelsea Finn, Stanford, co-founder of Physical Intelligence

Lecture 1 has 164,000+ views and is worth it if you want the big picture first.

→Behavior cloning, and why it fails when the robot drifts off the path it was shown
→Policy gradients and PPO, probably the most popular RL algorithm out there
→Q-learning and why it can learn from old data
→Offline RL: real robots are expensive to run, so most robot data is collected once and reused
→Model-based RL, and why an optimizer will exploit every mistake in a learned model of the world

Colab notebook coming soon

Install Stable-Baselines3, a free library with ready PPO and SAC. Then:

train PPO on a MuJoCo environment like Hopper and plot the learning curve
train the same environment with SAC and compare how many steps each needs
record 50 episodes of your Month 4 LQR cart-pole and train a small network to copy it with behavior cloning
push the cloned policy off its usual path, watch it fail, and explain why with AA203 lecture 15

You now have a controller you designed and a policy that learned from it. Comparing the two is a strong interview story.

Get the next checkpoint by email

One topic, one task and one new resource every week.