One in a Billion: How to Make Sure Autonomous Systems Are Safe (Enough)

Abstract

Autonomous systems, such as self-driving cars, are starting to pop up in the world around us. These systems use complex combinations of model-based and data-driven algorithms to make real-time, safety-critical decisions. In this talk, I will provide some thoughts (and hopefully insights) into what engineers need to do to convince ourselves that these systems are safe enough to release in to the real-world, where society expects that critical failures (those resulting in potential loss of life) will happen at rates of less than 1 in a billion. In particular, how do we make use of combinations of model-based simulations (even when we know the models are missing details that sometimes matter), log-based regression (even when the updated software won’t do exactly what was done in the logs), real-world operations (where you drive lots of boring miles and a few interesting ones), and structured testing (going beyond the 30 minute driving test, by which we decide 16 year-olds are allowed to drive any non-commercial vehicle, any place in the world, for the rest of their lives)? More specifically, I’ll talk about the role that formal specifications, assume/guarantee contracts, and test synthesis can play in helping address this challenge.