Waymo's Dmitri Dolgov: 20 Million Rides and the Road to Full Autonomy

- We have an unbelievable treat next. A founder who's touching a ton of lives, and who's been at it for a very long time. How many people here have been in a Waymo? Mmm. Okay, that's a relief. The chef eats his own food. I do, I do a lot of it. All right. And how many people love the Waymo experience? Rock on. I'm a daily active now. It's incredible. Excellent. Thank you. We have here the creator, a man who has been at this mission for Get this, founders who've been in AI since 2022.

Almost 20 years. building in the autonomous vehicle challenge. And he has not only been at this, he's been at it in the great times, at the tough times. He's been persistent and he has created something that is unlike anything else on Earth. Truly exceptional. Please join me in welcoming Dimitri Dolgov. Thanks. Great to be here. All right, Dimitri. So we've got about 25 minutes together. The goal is to understand a little bit about you. What makes you tick? What has made you persist since the early days of the DARPA challenge 21 years ago?

all the way through Waymo early days to today and the future. Let's start on you and then we'll get into technology very quickly. Sound good? Sounds good. Okay. So, Dimitri, you are known by your team as technically brilliant, incredibly intense, but also very kind and humble. You were born in the Soviet Union, raised in the States, and then chose to go back to one of the most prestigious intense physics programs on the planet in Moscow. How did those first few years of your life shape you? And how did it shape your character?

So my parents went to the same school so that to a large degree drove my decision to go back and went to high school. I actually traveled around quite a bit. I spent a year in Japan, then went to high school in the States and came back to college to do math and physics in Russia. So that was the same school that my parents went to. And I kind of really grew up hearing stories about what it is like to be at that place. I really wanted to go back. and the...

In those early days in college, one of the most important thing is-- Yeah. acquiring the abilities to learn and independently explore. So I think that really helped me in my future career. Now, you did this very intense program at Moscow Institute of Physics and Technology. And then you decided to keep going on the AI path. You earned your PhD also in AI. And then you pretty quickly were attracted to autonomous vehicles. In '05, you were part of the DARPA challenge. Can you tell us about those early days? What drew you to autonomy?

That was kind of a light switch moment for me. In the early days when I went to college and in grad school, it was more about the learning of the fundamentals. And I... didn't have at all a clear... picture, an idea of what I wanted to do after. And then I think the timing was just incredibly lucky that this was when I was finishing up grad school, the urban challenges, the grand challenge, and then the urban challenge, the one that I took place in, were happening, and it clicked. It was just the technology is incredibly interesting.

The mission is so powerful that nothing else come close, and it's a real product there. You can be hands-on, experience it yourself. it really checked all the boxes for me. And as you said, that's been 20 some years ago. Who's counting? And then I've never looked back, and that's what I've been doing Amazing. So Waymos started out of a project at Stanford automotive lab. There were two sides of this building. There was the autonomy side, and then there was a solar car side. Fun fact, I was an idealist. I worked on the solar car.

I got that bet very wrong. You bet on autonomy. Tell us about the first few years of Waymo from from 09 to the formative years. So we started in 2009. That was, at the time, the Google self-driving car project. And The first couple of years, it was... all about... learning the problem space, understanding what it means to try to put an autonomous vehicle on public roads. When we started in service of those goals of learning, understanding the problem space, we created a couple of goals for ourselves. One was to drive 100,000 miles total in full autonomy, which at the time was not heard of.

And the second one was to drive 10 routes. Each one was 100 miles long. They were all over the Bay Area, chosen to be very difficult. And we had to do each one from beginning to end in full autonomy, still with a person behind the wheel that can take control. But the challenge was to complete each one without an intervention. So it was a small team of us, it was about a dozen people. It was the early crazy startup days, everybody working 24/7, writing code and building hardware during the day, doing some testing at night.

And it took us about 18 months to complete both of those challenges. Incredible. It seemed impossible at the time. Now you guys are on hundreds of millions of miles. Absolutely. OK, so early Waymo days, extreme challenge, starting to achieve there. Next few years, You have a reputation in your team. for grinding really hard. You were sleeping at the office. Tell us about Dimitri in-- the first few years of Waymo and how you formed your leadership style. I got to say, those early days was probably the most fun I've ever had in my professional life.

And it is that that you know, momentum and that pace of an early... startup days, when you are making so much progress every hour of every day. And you're doing everything. You are working on-- setting up the hardware in the cars and then you know configuring and calibrating the sensors and you know, your pose estimation system. And then you're writing software during the day, and it's everything, right? It's the core of the, you know, the software, the algorithms that drive the car. It is all of the tools, you know, and UIs and the user experience in the car.

So you're doing everything. You're learning at an insane rate and you're making progress at an insane rate. So that was – those were the early days of Project Chauffeur. And then, you know, in those couple of years, we've convinced ourselves that – yes, this is worth pursuing. So we doubled down and started actually building towards the future of a fully autonomous product. OK, so exciting first few years. Intense, fast-paced, technically really difficult. Now take us to 2016-17 for a moment. This was a period where we actually had a hype cycle in AI.

Turns out there's been a few of them. AV, autonomous vehicle, was at the center of that hype cycle. I mean, I remember just so many companies going after this. And then there was a massive slump. And when most people gave up or failed or fell apart, you guys persisted. and you were a leader in that persistence. For all the builders in this room, how did you navigate through the hard times? Thank you. So first, a comment on-- Yep. what these cycles kind of look like to me and how I've seen them.

You said there's been many, some in AV, but more generally. And what often leads to a cycle like this is some Rapid. some breakthrough that leads to very rapid progress. in the early parts of the problem and very rapid investments still in the early part of the problem. And in the 80s, the problem has always had this property that it's very easy to get started, But it's very difficult to take all the way to a real product, full autonomy, and superhuman performance. So it's somewhat natural given those ingredients that whenever there's been breakthrough in technology, whether it's convolutional nets or transformers or large language models.

It's led to the cycle, "Okay, now the problem is going to be..." And it kind of reshapes the early part of the curve, but it doesn't change the long tail of it. Thank you. And-- for us, I think it was... understand. that It's... not going to be an easy problem, but it's a very important one. And believing in the Because today, Worldwide, somebody loses their life to your crash, on our roads every 26 seconds. So I guess it's the combination of knowing that the mission is really, really important. And then understanding what you're up against and not looking for easy wins or quick solutions or silver bullets that help the team have the right stamina to go the distance.

Brilliant. So, You guys were in this moment where it was really easy to get started. A lot of people got there, but you guys actually persisted. and got through to the other side with a truly magical experience. Pretty much every hand in this room went up. A truly magical experience because of that persistence. Let's talk about technology today. A lot of people are talking about world models. You have had all the components of world models for many years. How do you think about a world model? And what is Waymo's version of a world model?

Yeah, there are a few things that A few terms that people use nowadays. People talk about world models. world action models, omni models, visual language action models. And at the core of each, there's an ingredient that is relevant and really important for Waymo and for what we've been building in our AI ecosystem. at the core of our AI ecosystem is what we call the Waymo Foundation model, and it powers three... main pillars of our AI and our tech. It's the driver. The simulator? and the critic. And those are very related but distinct tasks.

So at the core of what our foundation model needs to be capable of, are things like it needs to understand how the world works, the physics, the dynamics of the physical world. Uh, And it needs to understand what it is to be a good driver and how the effects of the actions of that driver or eye agent on other agents in the world. And then we need to instantiate those in the physical agent that we're putting on the roads. So in a way, that foundational model that we've been building over the years is a multimodal world action language model.

It's multimodal in that it needs to be able to reason about not just images or video, but also other sensors like lighters and radars. model in that it really has to have a deep, precise understanding of the 3D spatial properties of the world, the dynamics, the physics, the behavioral aspects of other agents like cars, pedestrians, cyclists, and so forth. And we are not just passively modeling those worlds, we're an active participant in it. So we not only have to, you know, the world model has to be controllable, but also we need to have a deep understanding of what it means to be a good agent in that world.

And finally, it's aligned with language, and that allows us to kind of pull in the general world knowledge of a VLM into our model. That is very, very useful in giving us a boost in understanding the semantics and the deep social aspects of driving. And we've been, you know, working on productionizing that model for years, and really, it requires an extremely high degree of performance and accuracy and realism in every aspect of what we just talked about. Brilliant. So with this driver simulator critic architecture There's also been a lot of conversation about end-to-end architectures.

Is that the... appropriate dichotomy, how do we think about the approach to getting us to extreme performance efficiency, autonomous vehicles that are totally generalizable. As I'm very clear, the world model that is described, the Waymo Foundation model, is an So when we talk about an end-to-end model, we typically mean that it's one model that goes from sensors to decisions or actions. And There's some very nice properties of such a model. One of the most important ones is that you it learns the right rich representations between different components of the system, like the encoder and the decoder, or the perception and the planning part of your system.

as opposed to something where that interface is engineered, which is not sufficient for a task like driving. Now, I do think there's a false dichotomy there. There's end-to-end or something else. Really... in my mind, it's always been the question of, you know, it's end to end. And then, you know, what else? And what else do you need to build if you want to have a product that is fully autonomous, has superhuman level of safety, and you want to deploy at a scale and drive hundreds of millions of miles? And there it turns out that kind of the basic vanilla, you know, end-to-end system is insufficient.

Right. So, there's a massive difference between using end-to-end versus purely relying on it. So, at Waymo, we've really gone beyond that kind of basic vanilla end-to-end approach, and we've augmented the learned representation with structured, materialized, intermediate representation. And what that allows us to do are a few very important things that you might not actually need if you are building a different product, if you're building a a driver assist system, or a prototype, a demo, or a small scale deployment. But again, those things are absolutely critical if you want to go all the way to a fully autonomous, safe system with superhuman performance.

And those are things like having extra validation at runtime. in -- of the -- agent that's running on the car in the physical world is things like richer training and evaluation recipes that are very difficult or impractical to do in a pure kind of basic ATN system where this structured materialized representation gives you a boost in things like closed loop evaluation, closed loop training, rich reward functions for reinforcement learning. So that's been our approach. And all the human feedback that you get. from support and drivers dropping in and all of that, it's essential to have this type of architecture.

Exactly. Makes perfect sense. So not only have you innovated on the software stack, but also the hardware stack. There's a sixth generation now of Waymo driver, And you guys have always focused on being the driver. Tell us about the new sixth generation and what was it like the first time you've interfaced with it? So the six generation is our most advanced hardware suite, ancestor suite yet. The focus there has been on performance but also on simplification. drastic cost reduction and high-scale volume production. And this is the the driver that's powering our latest vehicle platform, that's the OHI.

earlier this year started fully autonomous operations. It's currently only open to employees, but coming to all of the riders later this year. Yeah, I... I had a chance to take a ride in one as soon as we started running fully autonomous operations. And, you know, the I spent a lot of my life in various generations of our cars. Every once in a while, there's a new first moment And that was definitely it. It's just that the coal car is designed around the rider experience. It is... Even though the external footprint of the car is about the same as the eyepiece, but inside you get in, it feels like it's a living room.

So much space in the back. We have new screens. We have these doors that slide open and will open automatically when you approach the car. So I had a blast and I can't wait to have this car in our fleet open to everyone. So you guys are going through a period of incredible scaling. For many years, you were in the lab, R&D purely. It took... 16 years-ish to get to 100 million miles, six months-ish to get to 200. Things continue to scale really rapidly. 11 cities now, many, many more. on the horizon.

Tell us, what is it like to scale a new city? And then tell us about your daily life with Awaymo. How do you use it as a creator? Okay. AND I THINK THAT'S A GOOD THING. Well, there's a lot. Okay, so exponential scaling. First of all, absolutely. It's been... phase transition for us and how we're scaling. So to give you a Couple of additional data points. It took us eight years from the day when we started our fully autonomous operations to the day when we had our service, our driver, providing rides to the public in four cities.

Earlier this year, just a few weeks ago, we launched four cities in one day. Thank you. Given the over 20 million fully autonomous rides. 10 of those million happened in the last seven months. So that's what exponential scaling looks like. Launching new cities, There's operational components show up. collect the data, characterize the environment, I mean, I validate the driver. A significant part of it is starting the conversation with the local communities because it's a new thing. It's a new product. So it's on us to earn the trust of the people there.

And then more often than not today, we're seeing that the driver is generalizing incredibly well. And it's just a matter of high fidelity, rigorous evaluation and validation before we deploy the fully autonomous product. And then, you know, we go from there. And then the last, what was the last part of the question? Oh, what is it? My daily life. It was a multi-part question, yeah. Waymo is how I get around nowadays. That's how I got here today. It was a great ride from Palo Alto up to San Francisco on freeways.

I, my family use it. I have three kids. They love Waymo. They... I think nowadays they get annoyed if on a rare occasion we have to be in a car that's driven by... myself or my wife or another human being. They're like, "Okay, what's going on there?" I feel the same way at this point. They love it. It's been part of their lives. For the entirety of their lives, when we are driving around, there's two things. There's only two things that get cold outs from my kids nowadays. It's doggies. And it's Waymos.

Nice. Okay. Probably similar amounts of cognition between those two. Okay, so let's talk about safety. One of the most meaningful, exciting parts of partnering with Waymo has been the fact that there's 1.19 million people a year on Earth that die in road accidents. This is life or death. And not only does it touch everyone in this room, But everybody has some connection who's been impacted by this. You have been about safety from the very beginning. And it's actually pretty hard. In a Silicon Valley where it's move fast and break things and see what happens, you guys have been incredibly patient with safety.

Can you tell us about a story that made it very real to you and how you keep that safety culture at Waymo. So the numbers you mentioned. that's what drives all of us at Waymo, and the status quo is not okay. We've kind of grown this over time, but challenging the status quo is – really important to everyone at our company. You're absolutely right that how you go about building a system like this is different from you might do in other areas and other fields and other industries where safety has to be the non-negotiable foundation and you have to build that into your everything that you do from day one.

Your model architecture, your training and evaluation recipes, the mindset of the team. It can be very tempting to focus on capability first and get to the 90% very quickly, but how you go about the first 90%. is a totally different problem for how you go about getting to your next N9s. So keeping that in mind and focusing on safety as the non-negotiable fundamental layer from day one is super important. And then, you know, today we're driving more than 4 million miles in full autonomy per week. God. and you see... a lot of events from the field and today we have the data over 170 million miles, fully autonomous miles, where we see that the Waymo driver is more than 13 times safer than a human driver when it comes to serious injury-causing collisions in the cities where we operate.

And you see that sort of superhuman safety behavior manifesting itself on the roads daily. I see examples of recently there was a little while ago there's an example that I saw of a person I think it was a young woman, on an electric scooter on the road. And then she lost control and tripped and fell right in front of the Waymo. And the Waymo driver showed superhuman accuracy and reaction time. I was able to, you know, swerve and break and everybody walked away. So it's things like this that, you know, I myself personally and the team, the whole team find very rewarding in terms of actually having a real impact on the safety of our roads.

And the scale that we're operating, that 13x reduction means that we are preventing a serious injury. every eight days. And that impact will just grow as we scale up. We're going to open the room to audience questions, a couple, in just a moment. But before we jump in, I heard a story about the LIDAR detecting or the radar detecting the footsteps of somebody behind a bus? Did that happen? And how does that work? Yeah, this was one of those moments where I was – positively surprised by the emerging capability of our system.

And the situation was-- This was, I think, in San Francisco. the Waymo driver was at an intersection. There was a bus that crossed, and we're sitting there waiting at a red light. So the bus crossed and stopped partially blocking the interstation-- so our light intersection. So then our light turned green. the Waymo driver started to proceed. And as it's proceeding, it detects A pedestrian-- on the other side of the bus. And... You can't see through the bus. It's not through... not radars, not cameras. You know, the windows are reflecting the people inside the bus.

And then, you know, it starts to react defensively. And sure enough, a pedestrian emerges from behind the bus. And then we're able to nudge around them. And everybody goes on their way. So when I saw that, I was-- It blew my mind. I'm not sure what's going on. I guess capable as superhuman as the way my driver is, it doesn't see through solid objects. So actually what turned out was happening is that our LiDAR, was bouncing the signal under the bus. and got a little bit of a sparse return from the movement of the person's feet under the bus.

And that was enough for the Waymo AI to not only detect that there's a pedestrian there, but also make a prediction about what's going to happen in the future, And keep everyone safe. Mind-blowing. Pretty unbelievable. We've got time for one question from the group. And if anybody has a key-- Jim, sorry, no free codes. Not at this one. Yes, Jim, please. Thank you. Does this work? Yes. I was just saying, congratulations on all you've achieved. It's really mind blowing. If you think about the next five to ten years, really focus on the business model, What are the milestones?

What happens in major cities? What's going to be different than where we are today? Just kind of walk us through your vision of the future. So we're heads down in execution mode. We've transitioned from intentional sequential de-risking of the driver and key parts of the business to rapid parallel global commercialization. That means deploying the Waymo driver in more places across the United States. And today we're in 11 cities operating fully autonomous and serving our riders. That we're going to expand in those existing places. new geographies, new cities. We're also expanding internationally.

Announced that this year, we'll plan to offer a service in London. and in Tokyo. So you will see us just accelerating that deployment, all in service of our mission. Good news to our team in London. Well, we covered a lot, Dimitri, from the very early days where you could get A lot of distance with not a lot of technology to then persisting through extremely hard times in autonomous vehicles and getting that extra mile. We talked about world models, driver simulator critic architecture. We got into the hardware, the sixth generation hardware, safety and scaling.

But most of all, I hope that we learned a little bit more about Dimitri, the man who's brought the magic that is Waymo to so many of us. And as I've gotten to know you more and more, I'm constantly struck not only by your brilliance and your persistence and performance, but also your humility. It says a lot for accomplishing this much. Thank you, Dimitri. Please join me in thanking Dimitri for all he does. Thank you. Thank you. And for many lives saved. Thank you.