Robotics and Autonomous Systems

Robotics is where AI meets physics. A robot has to sense a messy world, work out where it is, decide what to do and move its body, many times per second, with consequences that cannot be undone. The same stack runs warehouse robots, surgical assistants, agricultural machines, drones and self-driving cars. This lesson walks through that stack: the classical pieces every robotics engineer must know (estimation, SLAM, planning, control), the learning-based methods that are changing the field (imitation learning, sim-to-real, vision-language-action models), and how autonomous vehicles are classified, deployed and argued safe.

Intuition: driving to an unfamiliar address

Think about how you drive somewhere new. Your eyes and ears gather information (perception). You combine what you see with a map and your sense of motion to know where you are (localisation). You choose a route and, moment by moment, a path around the parked van ahead (planning). Your hands and feet turn that plan into steering and pedal movements, correcting continuously when the car drifts (control). Then you look again. Robots run the same loop, the perception–planning–control loop (often called sense–plan–act), typically at 10 to 100 Hz for planning and up to 1 kHz for low-level motor control.

The loop in detail

Classical systems build these as separate modules with well-defined interfaces, which makes them testable and explainable. End-to-end learned systems map sensor input more directly to actions and can capture subtle behaviour, but they are harder to verify. Most deployed systems in 2026 are hybrids.

Sensors

Sensor fusion combines these so that each covers the others' weaknesses: the IMU fills gaps between camera frames, radar confirms speed in fog, lidar anchors camera depth. Fusion requires accurate calibration (each sensor's position and orientation relative to the others) and time synchronisation; a 50 ms timing error at 30 m/s is a 1.5 m position error.

State estimation and the Kalman filter

Every sensor is noisy, so robots maintain a belief: a probability distribution over the state. The Kalman filter is the optimal estimator for linear systems with Gaussian noise. It alternates two steps. With state estimate $\hat{x}$, covariance $P$, motion model $F$, process noise $Q$, measurement model $H$ and measurement noise $R$:

Predict (motion makes us less certain):

Update (a measurement $z$ makes us more certain):

The Kalman gain $K$ weighs the measurement against the prediction by their relative uncertainties.

Worked example (1-D). A robot's predicted position is $\hat{x}^- = 10$ m with variance $P^- = 4$ m². A range sensor reads $z = 12$ m with variance $R = 1$ m². Here $H = 1$, so $K = 4/(4 + 1) = 0.8$. The updated estimate is $10 + 0.8 \times (12 - 10) = 11.6$ m, and the variance becomes $(1 - 0.8) \times 4 = 0.8$ m². The result sits closer to the more trustworthy sensor, and the variance is smaller than either source alone.

import numpy as np

dt = 0.1
F = np.array([[1, dt], [0, 1]])          # constant-velocity motion model
H = np.array([[1.0, 0.0]])               # we measure position only (e.g. GPS)
Q = np.diag([1e-4, 1e-2])                # process noise covariance
R = np.array([[4.0]])                    # measurement noise variance (2 m std)

rng = np.random.default_rng(1)
true_x = np.array([0.0, 1.5])            # start at 0 m moving at 1.5 m/s
x, P = np.array([0.0, 0.0]), np.eye(2) * 10.0
for k in range(100):
    true_x = F @ true_x
    z = H @ true_x + rng.normal(0, 2.0, 1)       # noisy measurement
    x, P = F @ x, F @ P @ F.T + Q                # predict
    S = H @ P @ H.T + R
    K = P @ H.T @ np.linalg.inv(S)               # Kalman gain
    x = x + K @ (z - H @ x)                      # update with the innovation
    P = (np.eye(2) - K @ H) @ P

print("true pos %.2f vel %.2f" % tuple(true_x))
print("est  pos %.2f vel %.2f | pos std %.2f m" % (x[0], x[1], np.sqrt(P[0, 0])))

Measurements have 2 m of noise, yet after 10 seconds the filter estimates position to within about 0.3 m (true 15.00, estimate 14.71) with a reported standard deviation of 0.62 m. It also recovers the velocity (1.43 against the true 1.5 m/s), which was never measured directly. For non-linear systems, practitioners use the extended or unscented Kalman filter, or a particle filter, which represents the belief with weighted samples and can handle multi-modal beliefs ("I am in one of these two identical corridors").

Localisation and SLAM

Localisation estimates the robot's pose within a known map, for example by matching lidar scans to a prior high-definition map. When no map exists, the robot must build one while localising in it: simultaneous localisation and mapping (SLAM). This is a chicken-and-egg problem, because the map needs the pose and the pose needs the map, and errors accumulate as the robot moves. The key correction is loop closure: recognising a previously visited place, which reveals the accumulated drift and lets the system correct the whole trajectory.

Modern SLAM is usually formulated as a factor graph: nodes are robot poses and landmarks, and factors are constraints from odometry, IMU, landmark observations and loop closures. Solving SLAM means finding the poses that best satisfy all constraints, a non-linear least-squares problem solved with sparse optimisers (libraries such as GTSAM, g2o and Ceres). Variants include visual SLAM (cameras, e.g. ORB-SLAM), lidar SLAM, and visual-inertial odometry, which fuses a camera with an IMU and is what phones use for augmented reality. Learned components increasingly supply features, depth and place recognition, while geometry still does the optimisation.

Motion planning

Planning finds a collision-free, feasible path from start to goal. Planners work in configuration space (C-space): the space of all robot configurations, such as $(x, y, \theta)$ for a mobile robot or seven joint angles for an arm. Obstacles become forbidden regions in C-space.

• Graph search on a discretised grid or lattice: Dijkstra, and A*, which expands nodes in order of $f(n) = g(n) + h(n)$, where $g$ is the cost so far and $h$ an admissible heuristic (one that never overestimates the remaining cost), which guarantees an optimal path.
• Sampling-based planners for high-dimensional C-spaces: RRT (rapidly exploring random trees) grows a tree toward random samples; RRT* and PRM (probabilistic roadmaps) add asymptotic optimality and reuse.
• Trajectory optimisation: refine a path into a smooth trajectory that minimises cost (jerk, time, energy) subject to dynamics and collision constraints.
• Model predictive control (MPC): repeatedly solve a short-horizon optimisation from the current state, execute the first control, and re-plan. MPC blurs planning and control and is standard in autonomous driving and legged locomotion.

import heapq

GRID = ["S...#....",
        ".##.#.##.",
        ".#.....#.",
        ".#.###.#.",
        "...#....G"]
ROWS, COLS = len(GRID), len(GRID[0])
find = lambda ch: next((r, c) for r in range(ROWS) for c in range(COLS) if GRID[r][c] == ch)
start, goal = find("S"), find("G")
h = lambda p: abs(p[0] - goal[0]) + abs(p[1] - goal[1])  # admissible on a 4-connected grid

def astar():
    heap, came_from, g_best = [(h(start), 0, start)], {start: None}, {start: 0}
    while heap:
        f, g, node = heapq.heappop(heap)
        if node == goal:
            path = []
            while node is not None: path.append(node); node = came_from[node]
            return path[::-1]
        if g > g_best[node]: continue                     # stale heap entry
        r, c = node
        for nb in [(r+1, c), (r-1, c), (r, c+1), (r, c-1)]:
            if 0 <= nb[0] < ROWS and 0 <= nb[1] < COLS and GRID[nb[0]][nb[1]] != "#":
                if g + 1 < g_best.get(nb, float("inf")):
                    g_best[nb], came_from[nb] = g + 1, node
                    heapq.heappush(heap, (g + 1 + h(nb), g + 1, nb))
    return None

path = astar()
print("length:", len(path) - 1, path)

A* finds a 12-step path through the gap in the walls. Here the path length equals the Manhattan distance, so the heuristic was exact along this route. Real planners add costs for proximity to obstacles, inflate obstacles by the robot's radius, and smooth the result into a dynamically feasible trajectory.

Control

The controller makes the robot follow the plan despite disturbances. The workhorse is the PID controller: with tracking error $e(t)$ between desired and measured state, the command is

where the proportional gain $K_p$ reacts to the current error, the integral gain $K_i$ removes steady offsets, and the derivative gain $K_d$ damps oscillation. LQR (the linear–quadratic regulator) computes optimal feedback gains for linear models with quadratic costs, and MPC handles constraints such as actuator limits explicitly.

Learning in robotics

Imitation learning

Behaviour cloning trains a policy $\pi_\theta(a \mid o)$ by supervised learning on expert demonstrations of observations $o$ and actions $a$, collected by teleoperation or kinesthetic teaching. It is simple and data-efficient, but it suffers from compounding error (covariate shift): small mistakes lead the robot into states the expert never visited, where its errors grow. DAgger (Ross, Gordon and Bagnell, 2011) fixes this by running the learned policy, asking the expert to label the states it actually reaches, and retraining. Modern recipes predict action chunks (short sequences of future actions) and use expressive generative heads, such as diffusion or flow-matching policies, to handle demonstrations with several valid ways to do a task. Low-cost bimanual teleoperation rigs such as ALOHA made collecting such data much cheaper.

Sim-to-real

Simulation offers unlimited, safe, parallel experience, which is ideal for reinforcement learning. Policies trained in simulation often fail on hardware because of the reality gap: mismatches in friction, masses, latency, sensor noise and visual appearance. Remedies:

• Domain randomisation (Tobin et al., 2017): randomise textures, lighting, masses, friction and delays during training so the real world looks like just another variation.
• System identification: measure real parameters and calibrate the simulator.
• Fine-tuning on real data, and learning residual corrections on top of a simulated policy.

Sim-to-real RL is now standard for legged locomotion: quadruped and humanoid walking controllers are commonly trained in massively parallel GPU simulators and deployed zero-shot. OpenAI's 2019 Rubik's Cube work with a robot hand was a landmark demonstration of heavy domain randomisation for dexterous manipulation.

Vision-language-action models

Vision-language-action (VLA) models extend vision-language models so that they output robot actions. A pretrained VLM brings web-scale knowledge of objects and language; fine-tuning on robot demonstrations teaches it to map camera images plus an instruction ("put the banana in the blue bowl") to actions. Published examples:

• RT-2 (Google DeepMind, 2023) introduced the VLA term, representing actions as text tokens in a VLM's vocabulary and showing transfer of web knowledge to new instructions.
• OpenVLA (Kim et al., 2024), an open 7-billion-parameter VLA trained on the Open X-Embodiment dataset.
• π0 (Physical Intelligence, 2024), which pairs a VLM backbone with a flow-matching action expert to produce smooth, high-frequency continuous actions across several robot bodies.
• Gemini Robotics (Google DeepMind, 2025) and NVIDIA's GR00T N1 (2025), aimed at general-purpose and humanoid robots.

The trend is toward generalist policies trained across many robot embodiments. As of November 2026 these models handle impressive tabletop and household manipulation in research and pilot deployments, but reliability, speed and safety guarantees still lag what factory and road deployment require. Most production robots still rely on classical stacks with learned perception.

Autonomous vehicles and SAE driving automation levels

SAE J3016 (latest revision April 2021) defines six levels. The key concepts are the dynamic driving task (DDT), meaning the real-time operational and tactical functions of driving (steering, braking, monitoring the environment and responding to objects and events); the operational design domain (ODD), meaning the conditions a feature is designed for (road types, speeds, weather, geography, time of day); and DDT fallback, meaning who responds when something fails.

The line that matters legally and practically is between Level 2 and Level 3: at Level 2 the human is driving even with hands off the wheel, while at Level 3 and above the system is driving while engaged. The levels describe features, not whole vehicles, and are not a ranking of quality.

Deployment as of November 2026

• Level 4 robotaxis are in commercial service. Waymo runs fully driverless paid ride-hailing in multiple US metropolitan areas; TechCrunch counted ten in February 2026 (including Phoenix, the San Francisco Bay Area, Los Angeles, Austin, Atlanta and Miami), and the company has continued to add cities and has announced plans for London. Tesla began offering some robotaxi rides in Austin with no safety monitor in the car in January 2026, starting with a small number of unsupervised vehicles within a larger supervised fleet.
• Level 3 in consumer cars has stalled. Mercedes-Benz Drive Pilot was the first Level 3 system certified in the US (Nevada and California, 2023), limited to mapped highways at low speeds, and Germany approved it at up to 95 km/h in the right-hand lane. In January 2026, however, Mercedes was reported to be dropping Drive Pilot from the facelifted S-Class in favour of a more capable Level 2 system, and BMW dropped its Level 3 Personal Pilot from the 7 Series facelift. The industry has shifted toward broad "hands-off, eyes-on" Level 2 systems.
• Level 2 systems, including Tesla's "Full Self-Driving (Supervised)", GM Super Cruise and Ford BlueCruise, are widespread. The driver remains responsible at all times.
• No Level 5 system exists.

Treat any deployment snapshot as perishable; check operators' and regulators' current publications before citing it.

Safety cases

A safety case is a structured argument, supported by evidence, that a system is acceptably safe for a specific use in a specific environment. It is usually organised as claims, arguments and evidence, often drawn in Goal Structuring Notation. For autonomous systems, evidence combines:

• Functional safety (ISO 26262 for road vehicles): hazard analysis, safety integrity levels, redundancy, and freedom from faults in hardware and software.
• Safety of the intended functionality (ISO 21448, SOTIF): hazards that arise even when nothing is broken, such as a perception model misreading an unusual scene. This standard matters most for machine learning.
• UL 4600: a standard for safety cases of fully autonomous products, requiring coverage of the ODD, of machine-learning limitations and of field monitoring.
• Testing evidence at scale: simulation of millions of scenarios, closed-course tests of edge cases, and on-road miles with statistical comparison of crash rates against human benchmarks, analysed with confidence intervals because severe events are rare.
• Operational controls: ODD enforcement (the system refuses to engage outside it), a minimal-risk manoeuvre such as pulling over safely, remote assistance, and incident reporting to regulators.

The safety case is a living document, updated as the ODD expands and as field data arrives.

Common pitfalls

• Bad calibration and timing. Most "perception bugs" in fused systems are extrinsic calibration or timestamp errors.
• Overconfident filters. Setting process noise $Q$ too small makes a Kalman filter ignore measurements and diverge.
• Testing inside the training distribution only. Rare events (unusual vehicles, construction zones, glare) dominate real-world risk.
• Behaviour cloning without recovery data. Policies trained only on perfect demonstrations cannot recover from their own mistakes.
• Trusting simulation results. Policies exploit simulator artefacts; validate on hardware early.
• Mislabelled autonomy. Calling a Level 2 system "self-driving" encourages misuse; regulators and courts take this seriously.
• Ignoring latency. A planner that takes 300 ms in a 100 ms control loop is acting on stale data.

In practice

Start with ROS 2, the de facto middleware for robot software, and a simulator (Gazebo, MuJoCo, NVIDIA Isaac Sim, or CARLA for driving). Get calibration, time sync and logging right before any learning. Use proven estimation (robot_localization, a factor-graph SLAM package) and planning stacks (Nav2 for mobile robots, MoveIt for arms), then add learned perception. For manipulation, collect teleoperated demonstrations and train imitation policies, or fine-tune an open VLA when you need language-conditioned generality. Use RL in simulation with domain randomisation for locomotion and contact-rich skills. For any safety-relevant deployment, define the ODD first, build the safety case alongside the system, and log everything so you can learn from the field.