These experts are racing to protect AI from hackers. Time is running out
Bruce Draper bought a new car recently. The car has all the latest technology, but those bells and whistles bring benefits — and, more worryingly, some risks. “It has all kinds of AI going on in there: lane assist, sign recognition, and all the rest,” Draper says, before adding: “You could imagine all that sort of thing being hacked — the AI being attacked.”It’s a growing fear for many — could the often-mysterious AI algorithms, which are used to manage everything from driverless cars to critical infrastructure, healthcare, and more, be broken, fooled or manipulated? What if a driverless car could be fooled into driving through stop signs, or an AI-powered medical scanner tricked into making the wrong diagnosis? What if an automated security system was manipulated to let the wrong person in, or maybe not even recognize there was ever a person there at all? As we all rely on automated systems to make decisions with huge potential consequences, we need to be sure that AI systems can’t be fooled into making bad or even dangerous decisions. City-wide gridlock or essential services being interrupted could be just some of the most visible problems that could result from the failure of AI-powered systems. Other harder-to-spot AI system failures could create even more problems.During the past few years, we’ve placed more and more trust in the decisions made by AI, even if we can’t understand the decisions that are reached. And now the concern is that the AI technology we’re increasingly relying on could become the target of all-but-invisible attacks — with very visible real-world consequences. And while these attacks are rare right now, experts are expecting a lot more will take place as AI becomes more common. “We’re getting into things like smart cities and smart grids, which are going to be based on AI and have a ton of data here that people might want to access — or they try to break the AI system,” says Draper.”The benefits are real, but we have to do it with our eyes open — there are risks and we have to defend our AI systems.”Draper, a program manager at Defense Advanced Research Projects Agency (DARPA), the research and development body of the US Department of Defense, is in a better position to recognize the risk than most. He’s spearheading DARPA’s Guaranteeing AI Robustness Against Deception (GARD) project, which aims to ensure that AI and algorithms are developed in a way that shields them from attempts at manipulation, tampering, deception, or any other form of attack.”As AI becomes commonplace, it becomes used in all kinds of industries and settings; those all become potential parts of an attack surface. So, we want to give everyone the opportunity to defend themselves,” he says.Fooling AI even if you can’t fool humansConcerns about attacks on AI are far from new but there is now a growing understanding of how deep-learning algorithms can be tricked by making slight — but imperceptible — changes, leading to a misclassification of what the algorithm is examining.”Think of the AI system as a box that makes an input and then outputs some decision or some information,” says Desmond Higham, professor of numerical analysis at University of Edinburgh’s School of Mathematics. “The aim of the attack is to make a small change to the input, which causes a big change to the output.”For example, you might take an image that a human would recognize as a cat, make changes to the pixels that make up the image, and confuse the AI image-classification tool into thinking it’s a dog.”This isn’t just a random perturbation; this imperceptible change wasn’t chosen at random.”
Desmond Higham
This recognition process isn’t an error; it happened because humans specifically tampered with the image to fool the algorithm — a tactic that is known as an adversarial attack.”This isn’t just a random perturbation; this imperceptible change wasn’t chosen at random. It’s been chosen incredibly carefully, in a way that causes the worst possible outcome,” warns Higham. “There are lots of pixels there that you can play around with. So, if you think about it that way, it’s not so surprising that these systems can’t be stable in every possible direction.” More
