Self-Driving Cars Are Faced With Too Many Moving Objects And Too Little Time

Have you ever played the game of what-do-you-spy when on a long driving trip that includes a child in the car?

It is a simple and convenient way to pass the time, keeping the youngster preoccupied, and simultaneously entails the possibility of boosting their cognitive skills as a budding toddler.

Here’s how the game goes in case you’ve never utilized it.

While the car is zipping along, the child looks outside the car to try and identify various objects that can be readily seen while the vehicle is underway. For example, perhaps while on a vacation ride through open country areas such as farmlands, you tell the child to spot any cows. This gets the youngster eagerly scanning the surroundings in hopes of spying one.

If the toddler is quite young, it could be that they aren’t exactly sure what a cow looks like and might mistakenly call out when they see a horse or a goat. Upon a gentle and definition-providing correction by an adult, the child begins to learn what constitutes a cow versus other animals. This continues for possibly hours upon hours, though the search for cows might become tiresome and repetitive, thus, you might then switch to spotting something else, maybe tractors or plows.

You can up the ante for children that are more advanced in age. While you are driving in a city environment, you might ask the child to spy anyone on a skateboard or possibly find someone that is holding an umbrella. If it perchance is raining, the umbrella search is a bit over-the-top since there are bound to be dozens of them and the game is not particularly taxing (plus, having the youngster continually yelling out that they spotted one can be nerve-wracking for the adults present in the car).

At an extremely young age, a child might not be able to do much in terms of differentiating anything that they see. Asking to look for a cow or an umbrella is beyond their existing comprehension, and so it is usually the case that you might resort to having the youngster find entire buildings or perhaps identify puffy clouds in the sky. Besides the child not being able to identify smaller and distinctive objects, they might also have a difficult time telling one object from another. Asking them to find a particular person in a crowd of pedestrians could exceed their mental acumen and the group of people seems just a blur of many rather than a collection of distinguishable individuals.

Why bring up this popular child’s game?

Because it offers some keen insights into an aspect that continues to stymy the advent of AI-based true self-driving cars.

Specifically, one of the greatest challenges for a self-driving car is the AI being able to figure out the various objects surrounding the car and that is within the realm of the driving effort. You might think this is an easy thing to do since, as a human driver, you do this nearly effortlessly. While in the driver’s seat, you are constantly scanning back-and-forth to spot the other nearby cars and those pesky pedestrians that might suddenly dart into the street.

You can pat yourself on the back for being quite cognitively astute that you can visually examine the scene, identify the objects that can be seen, and make a myriad of mental judgments about those objects.

Consider what happens as you drive down an everyday neighborhood suburban street.

You look down the street and can see that it is tree-lined and some houses front to the street. There are cars parked on the street and some cars parked in driveways. With a closer look, you can see that there are fire hydrants, telephone poles, light posts, street signs, and so on. If you were to add up all the objects that you can see, the number might surprise you. It is a tremendous cacophony of objects that are within just that one simple and calm street that you are driving on (imagine the torrent of objects when driving down a busy downtown city block).

A newbie driver such as a teenager that is learning to drive can be overwhelmed when undertaking the driving task. For them, the neighborhood street presents a veritable tsunami of objects that each need to be considered. Will that car in the driveway suddenly startup and back down into the street? Are those children playing in the front lawn of a house going to inadvertently dart into the roadway as they chase after a football?

A parent that is teaching their teenager to drive is likely focused on the higher probability concerns and can rapidly assess what is worthwhile for attention and what does not require attention. That pack of kids playing on the grass is far enough from the street that it is unlikely they would reach it in time to get into the path of the car. And those cars on the driveways do not have anyone in them and therefore the odds of any of those cars backing into the street are nearly nil.

That’s what happens for seasoned drivers, while newbie drivers have not yet figured out how to optimally size up situations and ascertain what is a potential concern versus what can be given minimal priority.

Switch gears and consider the driving task for an AI driving system.

Via the sensors such as cameras, radar, LIDAR, ultrasound, and other specialized equipment, the AI is collecting data that represents the driving scene. That data has to be examined and interpreted, noting what objects are out there. In addition to identifying the objects, the driving question is whether any of those objects are a potential threat to the car and whether the car is a threat to any of those objects.

One important aspect to realize is that today’s AI is not sentient, and it does not have any semblance of common-sense reasoning. In case you are shocked at this revelation, I urge you to take the media exaggerations about AI being superhuman or otherwise close to human cognitive capacities as malarkey and do not fall for the abundant puffery. Perhaps one day we’ll have that kind of AI, but not now, and not in the near future.

So, the point is that the AI driving system is churning through numerous computations to try and ferret out the data coming into the sensors and then attempting to discern what the roadway consists of. This is all mathematically taking place and requires extensive computer processing to undertake.

That’s why self-driving cars are typically chockful of sensors and lots of computer processors. The computer processing needed is extensive. In the earlier days of self-driving car experimentation, the trunks and oftentimes the backseats of the vehicle were completely occupied with computer processors. Nowadays, given the miniaturization of computers, the devices are generally smaller than they once were, and they thankfully provide faster processing speeds.

We’ve then established that there is a lot of computational effort needed to examine the sensory data and try to identify what objects exist in the surroundings, along with assessing the nature of those objects, such as whether they are moving or sitting still, whether they are coming toward the self-driving car or going away from it, and other vital particulars.

Here’s the rub.

Besides the challenges of doing this kind of object recognition, there is also the time factor that raises its ugly and unrelenting head.

Time is crucial.

Suppose the self-driving car is proceeding down the street at a speed of 15 to 20 miles per hour. As a rule of thumb, a car moving at that speed has a stopping distance of approximately 40 to 60 feet or so (that is composed of the physical stopping distance plus the distance involved during the decision making to stop the vehicle). In that case, if an object enters onto the street and within the existing path of the self-driving car, and that object is less than approximately forty to sixty feet ahead, the AI can certainly try to do an emergency stop of the car, but ultimately has the potential of striking that object nonetheless.

Meanwhile, realize that the vehicle is already in motion, meaning that it is moving forward at a rate of about 20 to 30 feet per second at the speed of around 15 to 20 miles per hour.

Let’s add things up.

The AI receives data from its sensors and attempts to distill the data and assume for sake of discussion that this took the on-board computers around one to two seconds to accomplish. There is the time needed too for figuring out what action to take, so let’s add one to two seconds for that computer processing time. The AI then has to invoke the car controls, such as slamming on the brakes, which might take let’s pretend around one to two seconds to achieve.

All told, from the instant the sensory data was collected, we have around 3 to 6 seconds of computer processing time that occurred, before taking driving action per se. In that 3 to 6 seconds of processing time, the self-driving car would have proceeded forward at the 20 to 30 feet per second, thus, it has now gone an additional 60 to maybe 180 feet in distance. Add to that the stopping distance once the brakes are engaged, and the distance traveled could be about 100 to perhaps 220 feet.

A child or dog that perchance ran into the street is the kind of “object” that we all certainly hope and expect an AI driving system to avoid hitting, and perhaps it is rather evident that the faster the computer can process the scene, the sooner a needed driving action can be undertaken.

It comes down to this: Self-driving cars are faced with a very daunting problem, namely that there are in essence way too many objects and way too little time to calculate all the permutations and possibilities, along with then taking the appropriate driving action to ensure safe driving.

Returning to the discussion about humans, I trust that this depiction of the effort required to drive a car might impress you about how adept humans are at handling the driving of a car.

It is almost a miracle, I assert, that there are about 200 million licensed drivers in the U.S. driving a reported 3.2 trillion miles annually, and yet there aren’t even more car crashes than there are now (as emphasis, no car crashes are good, let’s be clear about that, but the aspect that it takes a tremendous amount of mental effort to drive safely and avoid getting into car crashes makes what we all do as drivers an astounding feat, I submit).

Okay, the gist is that self-driving cars have a tough job.

This leads to the aspect of what can be done to deal with too many objects in too little time that is the bane of existence for self-driving cars.

Let’s unpack the matter and see.

Understanding The Levels Of Self-Driving Cars

As a clarification, true self-driving cars are ones that the AI drives the car entirely on its own and there isn’t any human assistance during the driving task.

These driverless vehicles are considered a Level 4 and Level 5 (see my explanation at this link here), while a car that requires a human driver to co-share the driving effort is usually considered at a Level 2 or Level 3. The cars that co-share the driving task are described as being semi-autonomous, and typically contain a variety of automated add-on’s that are referred to as ADAS (Advanced Driver-Assistance Systems).

There is not yet a true self-driving car at Level 5, which we don’t yet even know if this will be possible to achieve, and nor how long it will take to get there.

Meanwhile, the Level 4 efforts are gradually trying to get some traction by undergoing very narrow and selective public roadway trials, though there is controversy over whether this testing should be allowed per se (we are all life-or-death guinea pigs in an experiment taking place on our highways and byways, some point out, see my indication at this link here).

Since semi-autonomous cars require a human driver, the adoption of those types of cars won’t be markedly different than driving conventional vehicles, so there’s not much new per se to cover about them on this topic (though, as you’ll see in a moment, the points next made are generally applicable).

For semi-autonomous cars, it is important that the public needs to be forewarned about a disturbing aspect that’s been arising lately, namely that despite those human drivers that keep posting videos of themselves falling asleep at the wheel of a Level 2 or Level 3 car, we all need to avoid being misled into believing that the driver can take away their attention from the driving task while driving a semi-autonomous car.

You are the responsible party for the driving actions of the vehicle, regardless of how much automation might be tossed into a Level 2 or Level 3.

Self-Driving Cars And Time

For Level 4 and Level 5 true self-driving vehicles, there won’t be a human driver involved in the driving task.

All occupants will be passengers.

The AI is doing the driving.

What can be done to cope with the too many objects, too little time conundrum?

One obvious factor is to go slower when driving.

A car that is moving at around 20 mph needs 40 to 60 feet or so to stop, while a car moving at 60 mph requires about 200 to 300 feet to come to a halt (these are all approximations, plus you need to factor the roadway conditions such as the surface of the road, whether it is slick from rain or ice, etc.).

The slower a car goes, the more time there is to make a choice about the driving action and to analyze the potential of hitting something or getting hit.

Most parents have their teenager drive in a parking lot when first learning to drive. This not only keeps the newbie driver away from the hectic antics of being immersed in traffic woes, but it also keeps the speeds low, such as perhaps 5 mph or so. That allows for more reaction time and thinking time.

There are self-driving cars that are being used in retirement communities, offering a ride-sharing service for those that live in the neighborhood and want to get from place to place. These self-driving cars are usually being capped at low speeds, which certainly befits being on such roadways. Also, some delivery oriented self-driving cars take groceries from the store to someone’s house, staying on lower speed routes and not venturing onto freeways or highways.

The advantage of being in that kind of driving milieu is that the processing activity does not necessarily need to be as fast. The Operational Design Domain (ODD), a technical term referring to the scope of a self-driving car, entails driving at low speeds in those locales and thus eases somewhat the burden on the AI and the processors.

When you hear about a self-driving car that seems to be operating well while driving, you need to be aware and ask what kinds of speeds are involved. It is somewhat of a trick to have a self-driving car that can readily operate at low speeds since you do not know for sure that it can equally drive as well at higher speeds. That being said, please do not misinterpret this as suggesting that the self-driving cars at the lower speeds are somehow less significant. They deserve as much attention and accolades, but just bear in mind that it is not necessarily the case that they can instantly scale-up to operating at higher speeds (most do, but some do not).

Now that we’ve covered the notion of moving at slower speeds, it is a facet or restriction that does not especially obviate the issue of too many objects and too little time, plus there is the reality that cars-are-cars and people will expect self-driving cars to be able to go at the full range of everyday speeds.

Okay, consider what else can be done to handle too many objects, too little time difficulties.

You can try to reduce the number of objects.

That sounds odd since the number of real-world objects is whatever number there are that happen to be surrounding the self-driving car. You cannot wish away those parked cars on the street or those kids playing on the front lawn. They exist. They are real.

From a processing perspective, you can try to reduce the number of objects, which perhaps humans do, though it is hard to know whether people do so in their minds or not. For example, the fact that there are houses all along the street might not be worthwhile considering per se, since they presumably have almost nothing to do with the objects that might come into the street. In a manner of speaking, you can ignore the houses. You can potentially ignore the trees that line the street. These are all objects that do not require much attention as they are not directly pertinent to the driving scene.

You can try to strip down the volume of objects being given any mental weight or attention. Fewer objects tend to mean less processing time needed.

Speaking of processing, you can speed-up the processors that are on-board the self-driving car by putting in place faster and faster hardware. The faster the processors, the more processing they can presumably do in any given segment of time.

You can improve the AI software in terms of how it ascertains the potential intersecting of objects and the self-driving car. Mathematically and computationally, the more efficient those algorithms are, the less processing they require, and the less time they tend to consume.

And so on.

Conclusion

Perhaps one of the biggest splashes in the news about the processing speeds for on-board computers has often gone to the periodic updates about Tesla Autopilot, and as noticeably amplified by the customary outsized remarks of Elon Musk (see my coverage at this link here), including his declarations about so-called 4D and the purported Full Self-Driving (FSD) aspects.

Please be aware that all of the self-driving car automakers and tech firms are continually in the midst of upgrading their hardware and software, partially to contend with this too many objects and too little time dilemma, even though they perhaps don’t get quite as large a media frenzy when doing so. Plus, the companies such as NVIDIA that make the hardware and software are also continually pushing ahead on speeding up their wares (see my coverage at this link here).

The need for speed is an intrinsic element underlying self-driving cars, whether it is the speed of the car itself or the speed of the processing or the speed of analyzing the driving scene and making crucial driving choices.

It is going to take time to find and optimize the best ways to make use of time, ensuring that AI-based true self-driving cars are both timely and safe.

Time will tell and the clock is ticking.