Autonomous vehicles like Waymo’s robotaxis navigate using three sensor systems working together. Cameras provide visual detail including traffic lights, signs, road markings, and pedestrians. Radar measures distance and speed and performs well in difficult conditions. LiDAR fires millions of laser pulses around the vehicle, measuring return times to build a precise three-dimensional picture of everything nearby. Before operating in a new area, Waymo builds detailed maps of lane markings, curbs, crosswalks, and signals, and the vehicle matches live sensor data against those maps to locate itself precisely.
Daniel assumes robotaxis are just following GPS. The correction opens a conversation that moves through the four questions the car is constantly answering, where am I, what is around me, what might happen next, what should I do, then into how the system predicts other road users’ movements, why robotaxis operate in defined areas, and what engineers mean by the long tail of rare situations. Waymo has now completed more than twenty million fully autonomous rides.
DanielMom, I saw a video of a Waymo RoboTaxi driving through San Francisco with nobody in the front seat. How does it actually know what to do? Is it just following GPS?
MomDaniel, GPS is part of it, but only part. The car is constantly figuring out where it is, what is around it, what the people and vehicles nearby might do next, and what it should do in response. All in real time, without a human driver.
DanielWow! It is doing all of that at the same time?
MomDriving is extraordinarily complex. We just don't notice, because we have been doing it since we learned to walk. But think about everything your brain processes when you cross a road. A RoboTaxi has to do all of that and more.
DanielBut how does it know exactly where it is?
MomBefore Waymo operates in an area, it builds extremely detailed maps, lane markings, curbs, crosswalks, signs, and signals. Then, while the car is moving, it compares what its sensors are seeing with that map. So GPS can help, but the car is also locating itself by recognizing the world around it.
DanielHmm, so how does it exactly see?
MomThere are three main kinds of sensors working together. First, cameras. Cameras around the vehicle give it a continuous view of the road. They can detect traffic lights, lane markings, signs, pedestrians, and cyclists. But a camera is only one way of sensing the world.
MomSo RoboTaxis combine cameras with sensors that are especially good at measuring distance, speed, and three-dimensional shape.
DanielSo, I guess cameras alone are not enough.
MomWhich is why RoboTaxis also use radar, sending out radio waves that bounce off objects and return. Radar is excellent at measuring how fast something is moving and works well in bad weather. And then there is LiDAR, light detection and ranging. The car fires millions of laser pulses in different directions around the vehicle.
MomEach pulse bounces back, and the system measures exactly how long it took.
DanielI see, and from the return time, you can calculate exactly how far away everything is.
MomAnd use those measurements to build a precise three-dimensional picture of the environment around the car. Every other vehicle, every pedestrian, every wall, every lamppost, located in three-dimensional space and continually updated. LiDAR gives the system extremely precise information about depth and shape.
DanielSo, I judge distance with my eyes and brain, but the RoboTaxi can actually measure the three-dimensional space around it directly.
MomAnd all three systems, cameras, radar, LiDAR, feed their data into software simultaneously. This is called sensor fusion. The car is not relying on any one sensor. It is combining everything, so that the other sensors can provide complementary information.
DanielSo, how does the car learn what all of this means, and what might happen next?
MomThe software has learned from enormous amounts of driving experience, both on real roads and in simulation. Machine learning systems help it recognize road users, and understand patterns in how they move. A pedestrian approaching a curb. A car drifting toward another lane.
MomA cyclist preparing to turn. And the system does not just ask what is there. It estimates what might happen next.
DanielSo, the car is constantly asking, what is everything around me about to do next?
MomAnd, planning a response before it happens. The system considers multiple possible futures. What if that pedestrian steps off the curb? What if that car breaks suddenly? And uses those possibilities to choose a safe path forward.
DanielSo, it is not just reacting. It is also anticipating.
MomWhich is actually closer to how a skilled driver thinks, than how a nervous beginner drives. Beginners react to what is happening. Skilled drivers are always reading what is about to happen.
DanielIf it is so good at all of this, why are robo-taxis still only in certain cities? Why haven't they taken over everywhere?
MomBecause the real world has what engineers sometimes call, a long tail. Rare and unusual situations, that are difficult to anticipate and test. A traffic officer giving unusual directions. Debris falling into the road. Strange construction. An unpredictable driver doing something nobody expected.
MomEngineers have to make sure the system can respond safely. Not just to ordinary driving, but to an enormous range of unusual situations too.
DanielSo the car might handle a million situations perfectly, and then encounter one it was not ready for.
MomAnd the stakes of getting it wrong are high. So systems like Waymo operate within defined conditions, and places where they have been extensively tested, and validated. Engineers use real world driving, testing, and enormous amounts of simulation, to improve the system, and prepare it for unusual situations.
MomBut a car that can autonomously drive anywhere on earth, in every road and weather condition, that is still an open challenge.
DanielSo a robo-taxi is not really a car that can drive anywhere. It is a car that can drive very well, in the specific places it has been prepared for.
MomWhich is still extraordinary. A few years ago, that sentence would have sounded like science fiction. Waymo has now provided more than 20 million fully autonomous rides. Driverless cars are carrying real passengers through real cities every day, within the places and conditions the system is designed to handle.
DanielA car. No driver. Firing millions of laser pulses around itself, predicting what might happen next, in real cities, with real traffic.
MomAnd somewhere in San Francisco right now, someone is sitting in the back seat, reading a book, while the car handles everything. So Daniel, what is one thing you are taking with you today?
DanielSo, let's see. A robo-taxi first needs to know exactly where it is, using detailed maps, GPS, and what its sensors can see around it. Then, cameras give it visual detail, radar helps measure distance and speed, and LiDAR uses laser pulses to build a precise, three-dimensional picture of the world around the car.
DanielAlso, the software combines all of that information to identify what is around it, predict what might happen next, and plan what the car should do. Oh, and also, one of the hardest problems is the long tail. All the rare and unusual situations that engineers have to make sure it can handle safely.
MomA car that measures the world around it considers what might happen next, and chooses a path through it.
DanielI want to sit in the back seat and watch it handle a roundabout.
MomThat would be a very satisfying thing to watch.
