Measurable Safety, The Missing Ingredient To Demonstrating ADAS Value

“If you can’t measure it, you can’t improve it,” said renowned management guru Peter Drucker. Drucker meant that one cannot declare success without defined metrics. With metrics, one can track progress and adjust development to produce positive movement. Without them, one is always in a haze of doubt. Interestingly, the lack of clear metrics not only hurts the producer, but without these metrics, the haze of uncertainty causes the buyer to impose a discount on the product’s value. 

Hard metrics are not new to the automotive industry. Fuel efficiency (MPG) or performance(zero to sixty) are just some of a whole host of hard metrics published and tracked by the industry. However, ADAS safety systems are one place where there is a remarkable lack of clear metrics. Not only are there no hard metrics, there is such confusion that that recently a group of consumer advocated (AAA, JD Power, Consumer Reports, and National Safety Council) felt the need to publish a common naming structure consisting of the four major categories (driving control assistance, collision warnings, collision intervention, and parking assistance) of ADAS functionality.  

You might ask, where are the regulators ?

To date, regulators have taken a laissez-faire attitude towards ADAS. As “AV 4.0: Reasonable For AV, But Perhaps Missing The Boat On ADAS Regulation” discusses,  AV 4.0 does not offer any guidance on ADAS. This is despite the fact that systems such as driving control assistance and collision intervention can actively engage in the driving task.  

What about Congress ?

Indeed Congress has been working on a bipartisan bill on autonomous vehicles. As “New AV Bill, Its Bipartisan, Is It Better ?”  explains, there are many fine focus points to this bill, especially on the topic of cybersecurity, but it does not address ADAS regulation. Perhaps the reason for this lack of concern is that these systems just work. So, how well do they work ? The Insurance Highway Safety Institute (IIHS) has done some studies of the effectiveness of ADAS systems, and found that tasks such as “active lane-keeping” were a challenge for the tested commercially available automobiles, and conflict avoidance systems are getting better.  Thus, it appears that there is some cause for concern.

What is the result of the lack of hard metrics ?

1. Customer Value:  As “New car buyers unwilling to pay for advanced safety features” explains,  the first predictable result is that the customer does not value the capability, and will not pay much incrementally for it. It is hard to pay for something where the value is: ”we will try to help you in an undefinable manner, but ultimately you are on your own.”     
2. Insurance Risk:  As “Insurers say no discounts for ADAS” explains, insurers are refusing to provide safety discounts for ADAS.  The original users of data mining, they claim that data does not yet support such discounts. 
3. Maintenance Cost: ADAS consists of sophisticated sensors which require care in terms of maintenance, and from a consumer point-of-view, this can be a “hidden” cost.  In fact, this situation has become sufficiently problematic that consumer reports put out a bulletin warning customers of this issue. 

Where does this leave ADAS ?  A feature which consumers do not highly value, insurers are not sure it adds value to safety, and by the way, it has hidden costs. Finally, the ultimate liability for ADAS has not been litigated through the court system yet. The conventional wisdom is that the driver carries the liability, but just one accident of premature breaking in collision intervention can remarkably change the liability picture. The vague expectations of current ADAS systems invite lawsuits.

Thus, it seems there is a dire need for some metrics which define function and expectation clearly.  What might this look like ?   Let’s consider the case of radar based conflict avoidance systems:

1. Curvature, Hills, and Valleys:  The expectation is that the conflict avoidance should work independent of the curvature of the road or whether you are on a hill or valley. Is this true ?  Where is the evidence ?  
2. Road Debris:   Should conflict avoidance be triggered based on road debris ?  If so, what exactly would trigger it ?  What should NOT trigger it ?
3. Driver Attentiveness:  Should conflict avoidance be triggered based on driver attentiveness ? If so,  what exactly constitutes driver attentiveness ?
4. Physical Environment:  The expectation is that conflict avoidance will work independent of the surrounding environment. It should not matter if you are in a tunnel or under a bridge.  Yet, we know radar based solutions are susceptible to the “ghost car” problem due to reflection and interference. Where are the metrics which show the ability to address these problems?

Some underlying technology and standardization efforts are forming to address these topics.  For example, the UN’s effort with ALKS  starts to look at the lane keeping systems. Also, standardization bodies such as the Association for Standardisation of Automation and Measuring Systems are building a modeling framework with ASAM ODD which can reasonably model the interesting scenarios. These two pieces form the underlying components upon which to build reasonable engineering metrics. 

Overall, it would seem wise for the automotive industry (and regulators) to build a framework for measurable safety for safety critical ADAS systems such as conflict avoidance and active lane keeping systems. The metrics could be detailed at the engineering level or a higher level figure of merit (such as the SAE Levels of Automation) which captures broad tiers of demonstrated safety. Without this work, it is hard to see how the automotive industry can capture a return on the investments being made for the development and deployment of these systems.