Tesla and Mobileye autopilots duped by “phantoms”

It’s a common movie plot device, the main character thinking they saw someone step onto the road, so they swerve and end up in a ditch. Now imagine it’s real — sort of — and instead of a trick of the light or the mind, that image comes from a cybercriminal projecting, for a split second, something the car autopilot is programmed to respond to. Researchers from Georgia Tech and Ben-Gurion University of the Negev demonstrated that sort of “phantom attack” threat at RSA Conference 2021.

The idea of showing dangerous images to AI systems is not new. Techniques usually involve using modified images to force the AI to draw an unexpected conclusion. All machine-learning algorithms have this Achilles heel; knowing which attributes are key to image recognition — that is, knowing a bit about the algorithm — makes it possible to modify images so as to hinder the machine’s decision-making process or even force it to make a mistake.

The novelty of the approach demonstrated at RSA Conference 2021 is that the autopilot was shown unmodified images — an attacker need not know how the algorithm works or what attributes it uses. The images were briefly projected onto the road and nearby stationary objects, with the following consequences:

In a variation on the theme, the images appeared for a fraction of a second in a commercial on a billboard by the side of the road, with essentially the same outcome:

Thus, the authors of the study concluded, cybercriminals can cause havoc from a safe distance, with no danger of leaving evidence at the scene of the crime. All they need to know is how long they have to project the image to fool the AI (self-driving cars have a trigger threshold to reduce their likelihood of producing false positives from, for example, dirt or debris on the camera lens or lidar).

Now, a car’s braking distance is measured in dozens of feet, so adding a few feet to allow for better situation assessment wasn’t a big deal for AI developers.

Length of time required to show a phantom image to Tesla and Mobileye recognition systems. Source

However, the figure of a couple of meters applies to the Mobileye artificial vision system and a speed of 60 km/h (about 37 mph). In that case, response time is about 125 milliseconds. Tesla’s autopilot response threshold, as experimentally determined by the researchers, is almost three times as long, at 400 milliseconds. At the same speed, that would add almost 7 meters (about 22 feet). Either way, it’s still a fraction of a second. Consequently, the researchers believe such an attack could come out of the blue — before you know it, you’re in a ditch and the image-projecting drone is gone.

One quirk in the system inspires hope that autopilots will ultimately be able to repel this type of attack: Images projected onto surfaces that are unsuitable for displaying pictures are very different from reality. Perspective distortion, uneven edges, unnatural colors, extreme contrast, and other oddities make phantom images very easy for the human eye to distinguish from real objects.

As such, autopilot vulnerability to phantom attacks is a consequence of the perception gap between AI and the human brain. To overcome the gap, the authors of the study propose fitting car autopilot systems with additional checks for consistency in features such as perspective, edge smoothness, color, contrast, and brightness, and ensuring results are consistent before making any decision. Like a human jury, neural networks will deliberate on the parameters that help distinguish real camera or lidar signals from a fleeting phantom.

Doing so would, of course, add to systems’ computational load and effectively lead to the parallel operation of several neural networks at once, all necessarily trained (a long and energy-intensive process). And cars, already small clusters of computers on wheels, will have to turn into small clusters of supercomputers on wheels.

As AI accelerators become widespread, cars may be able to carry several neural networks, working in parallel and not draining power, on board. But that’s a story for another day.