Month 3 · ~17 h of video + practice

How a robot sees and senses

Your goal this month: understand how cameras, IMUs and lidar turn the world into numbers, and how neural networks read those numbers.

A robot that can't see can only repeat the same motion forever.

Perception is the layer where machine learning first took over robotics, and it's still where most robot learning starts.

month 30/7 · 0%

Watch

Fei-Fei Li, Justin Johnson and team, Stanford

One of the most famous deep learning courses in the world, fully up to date with transformers and robot learning.

Lectures 1 to 6: classification, neural networks, backpropagation, CNNs
Lecture 8: attention and transformers
Lecture 9: detection and segmentation
Lecture 15: 3D vision
Lecture 17: Robot Learning, by Yunzhu Li. Save it for the end of the month, it's the bridge to everything that follows

Stanford

The course most robot learning people never watch, and why they're surprised when a real sensor doesn't behave like the simulator.

→How a CNN turns pixels into a prediction, and what backpropagation actually does
→Why transformers took over vision
→Detection and segmentation: a robot needs to know where things are, not only what they are
→Sensor noise and drift, and why a robot's estimate of where it is gets worse over time
→The difference between what a camera sees and what a lidar sees

Colab notebook coming soon

Add a camera to your MuJoCo scene from Month 2 and place three colored objects on the table. Then:

render images from the simulated camera and train a small CNN in PyTorch to find one object's position
use your Month 2 inverse kinematics to move the gripper above the predicted position
add random noise and lighting changes to the images and watch the accuracy drop

That last step is your first taste of the sim-to-real gap: clean simulated data versus messy real data.

train and evaluate a CNN in PyTorch
explain attention in one paragraph without equations
explain why an IMU drifts and why lidar and cameras fail in different ways
connect a vision model to a simulated arm, so it sees an object and moves toward it

Get the next checkpoint by email

One topic, one task and one new resource every week.