Most Self-Driving Demonstrations Are Theater, Here’s How To Make Them More Real

Almost every company producing self-driving car technology has released a video showing their car taking a drive in some situation. The videos vary from short to long, and the situations vary from simple to complex. They are meant to show off the capability of the team. And what do you know, all the ones released directly by teams are all flawless. The lack of mistakes is no accident, though.

Instead we need to see either the results of use by the public on routes of their own choosing, or failing that, a deliberate video done at a randomly selected time on a randomly selected route, announced in advance and filmed live.

The existing videos do show something basic — the team has reached the professional level to pull off a drive and accomplish certain tasks. Early in their lives, teams can’t go an hour without a mistake in any complex environment. We don’t expect them to show off, and they can’t. But Google Car (Waymo,) where I worked at the beginning of this revolution, produced some videos after just a year of operation.

Above is an example video from one of the players — Intel/MobilEye. Their one hour video, which they make a big point of saying is unedited, shows they can pull off an hour, and shows situations it can handle some of the time, but doesn’t tell us if they could do that in a random test.

There’s no need for an hour here. The duration reveals little — the video would demonstrate more by being just the highlights of the individual capabilities shown off.

The problem is that generally, the videos are cherry picked. In the extreme, they might make many videos until they get one without any problems. More likely they just take a route that they have driven many times before and know the car handles well, so that the car will handle it on the first video, or perhaps the second. They are not overly trying to deceive, but rather to show off what they have done.

Consider this Tesla video from 2106, part of the system which got Elon Musk talking about how their system would be “feature complete” not long after that. The team was no doubt pleased and proud to make this video, but we know today the car, even 5 years later with much more advanced hardware and software, isn’t actually nearly this good.

We know that because Telsa is now doing it mostly right, allowing random users to take videos of streets they choose. While those videos show the cars frequently need interventions (as is not surprising for a test version of a driver assist system, in spite of its inaccurate “full self driving” name) they show that because we are not seeing a pre-selected demonstration. (Well, sort of. A recent disclosure reveals that many of Tesla’s beta-testers were under NDA, and under an admonition to avoid publishing videos that made Tesla look bad. I don’t think this could have been a very strong NDA or strong admonition, considering what was shown, but it’s still concerning that it was there.)

Letting the public ride in the car, particularly alone with no NDA on routes of their own choosing, is a real demonstration. Tesla has mostly done this, and Waymo has done it. A number of other teams operate limited robotaxi services with a safety driver on board, though usually over fairly limited routes. In China, some unmanned robotaxi operation is underway as well. If there is objective reporting of randomly selected rides, that can be a measure of the level of the product.

Many teams are not at the level of being able to do this, but they still want to convey how far along they are, and the public wants an honest demonstration of this. So here’s how to do it:

1. Announce you will do a demo on a specified day, an ordinary day with average traffic. Publish in advance a series of a large number of possible routes the vehicle can drive, such that the starting point can be reached in a modest time by some vehicle.
2. At the appointed time, a random route will be chosen through some transparent random process.
3. The vehicle will drive to that route (human driving is fine for this) and immediately self-drive it, streaming the drive live, as well as recording it in better quality. A safety driver can be present, but of course if they intervene, this should be obvious on the video.
4. Repeat this on different days with different random routes, possibly at different times of day selected the day before at random.

For a trucking firm, this random test is difficult. A truck system will be limited to freeways. From any one location there will only be a handful of routes it can drive. If they have many trucks they could place them in many places to allow the random test. Otherwise they would publish a list saying, “Drive these 200 miles of interstate at time X” where the route is reachable by their truck at time X.

If one can do this several times without interventions, this is a demonstration that the system can probably drive that long over a variety of routes without intervention. However, “that long” is still just the duration of the videos. A real robocar has to drive for around 3,000 hours (120,000 miles) without any contact incidents to begin to match the poor level of the average human driver, and 12,000 hours without a police reported accident. That’s a bar that can’t be shown in any number of videos that people could watch, and requires an extensive pilot test with lots of data. Of course, even 12,000 hours just starts to show the quality — to actually get a firm demonstration requires much more, which only Waymo has yet done.

Read/leave comments at this page