Volvo Trucks Wants To Know If You're Going To Cross That Street

Getting an autonomous vehicle to react to someone darting out in front of the car means collecting as much information as possible about the conditions on the road ahead. The various sensors, cameras and connected devices in an autonomous vehicle (AV) can all analyze the situation and determine what the car should do: brake, turn or speed up. But what if the AV could recognize something that hasn’t even happened yet?

That’s the question that Volvo Trucks North America and Perceptive Automata are trying to answer with a project they are undertaking with Dependable Highway Express (DHE). Perceptive Automata is working on artificial intelligence (AI) software that assesses the intention of nearby pedestrians – and, in the future, cyclists and other drivers – by teaching the machine to understand the same sort of subtle visual cues that humans use, like where a person is looking, how they’re waking and their body language. Perceptive Automata CEO Sid Misra said the technology is like giving truckers an extra set of eyes.

Misra spoke at a recent showcase for the technology at DHE’s headquarters in Ontario, California, where Perceptive Automata and Volvo Trucks conducted live demos using specially equipped XC90 SUVs. The test vehicles were driven on nearby public roads and, when they encountered pedestrians, their AI systems instantly categorized the people using two metrics: intention and awareness. Here, intention is the system’s expectation that the person will move into the street while awareness is the likelihood that the pedestrian is aware of the AV. The demos were short, but looking in on what the AI was doing proved that it could reliably assign nearby people the correct results of these two factors.

Perceptive Automata grew out of Harvard and MIT and was originally interested in AI that could predict human behavior in a number of areas, like security and sales environments. The mission has since changed to focus only on autonomous vehicles (for now) and that attracted Volvo’s attention. In July 2018, Volvo and Perceptive Automata installed data kits on two DHE trucks and collected over 100 hours of data in August and September. October and November were spent fine-tuning the model, then analyzing the data with the help of actual humans to create the training data.

“The magic is in how we give humans 100 hours of drive recordings and images of pedestrians and cyclists and drivers and elicit from them the right responses so we can turn that into training fodder,” said James Gowers, Perceptive Automata’s vice president of strategy and business development. Those 100 hours were enough to get the system working in the prototype trucks, but Gowers said that Perceptive Automata’s next big challenge is getting enough real-world learning into their AI. “We need more data, we need more edge cases,” he said.

That’s because Perceptive Automata wants the AI to be able to determine the intent of bicyclists and other cars as well as pedestrians, Gowers said. The underlying principles are the same in all of these cases, but there are differences in the questions that the engineers will ask the human assistants to the AI, focusing on whether a car is going to change lanes, for example, instead of if a person will cross the road.

Perceptive Automata’s technology only needs cameras at this point, but using LiDAR and thermal cameras are  being considered to allow the technology to work in awful weather. “It is surprising how much you can pick up with thermal imaging, in terms of body language,” Gowers said. “We can do a lot with cameras only, but if you are doing a full self-driving system you do need more than just cameras. You need LiDAR, you need thermal imaging, you need radar.”

While this particular project involves Volvo Trucks, Perceptive Automata is not tied to just one automaker. In October 2018, Perceptive Automata announced it had raised $16 million in Series A funding, including investments from Toyota AI Ventures and Hyundai CRADLE, proving that the auto industry is paying attention. So far, Perceptive Automata has raised over $20 million.

Aravind Kailas, research and innovation manager at Volvo Group North America, said that the way Volvo is working with Perceptive Automata on this project gives the global company an opportunity to try new things in a short amount of time, even without having any firm commitment to the automaker.

“Word is spreading internally through the Volvo Group, people are realizing that it’s not just trucks but buses, for example, or construction equipment operating in urban environment,” Kailas said. “As we started engaging with the startup, we realized that it’s absolutely critical to have this as a component [of AVs].”