Hostile takeover: On the dangers of prompt injection and delegating decision-making to AI
Our interactions with AI systems are through prompts—instructions that we provide the AI to get what we want. But what if it acts on prompts that we never gave it in the first place? And what if such malicious prompts result in serious consequences for businesses?
That's the danger of prompt injection in a nutshell. As AI systems become autonomous and capable of executing tasks without explicit instructions, the dangers of prompt injection attacks also become greater. OWASP Gen AI Security Project, an open-source initiative to identify risks associated with generative AI technologies, identified prompt injection as AI's most critical security risk. From the report:
A Prompt Injection Vulnerability occurs when user prompts alter the LLM’s behavior or output in unintended ways. These inputs can affect the model even if they are imperceptible to humans, therefore prompt injections do not need to be human-visible/readable, as long as the content is parsed by the model.
To find out the adverse effects of prompt injection on privacy and security, and how businesses can navigate AI systems without stepping on cybersecurity landmines, we spoke to Karuppaiah Veeraiah, head of cyberdefence at Zoho Corporation. With over 25 years of professional experience, his expertise spans security operations, vulnerability research, and threat detection. He's also the director of technology at Ulaa, a privacy-focused web browser which he helped build from scratch.
Below are edited excerpts from the interview:
Based on your experience building Ulaa, a browser that puts privacy and security at the forefront, what kind of privacy risks that you see are inherent in AI browsers for businesses?
Prompt injection is the primary problem. In AI browsers, there's a possibility that the content the AI processes comes from a third party. This third-party content, which can instruct the AI to execute certain tasks, can be completely invisible. For example, it might be white text on a white background that a human won't catch in an email. But the AI reads it and carries out whatever it's been instructed to do.
Historically, in computing, there's always been a problem differentiating between the code running the program and the data being processed. That's the basis of any injection attack. The AI browser is going to browse content from the internet and make decisions based on the content it's surfing. It's going to process everything. Because of that, command injection is quite possible.
How would you explain to a small business owner, who sees AI as a way to get their daily tasks done quicker and might be willing to use AI browsers, the dangers of prompt injection? What's the worst that could happen?
Let's take an autonomous car as an example. Since it's an autonomous car, it operates purely based on the map. But assume that map has been manipulated by someone else. What happens then? The car will drive you to a completely different destination, and you won't even realize it because you trust the vehicle. You trust the car to take you where you want to go, and the autonomous system is doing exactly what it's supposed to do—except it's blindly believing the map. You trust the car, the car trusts the map, but the map is manipulated. You end up at the wrong destination.
Now, assume you've automated your outward payments to vendors using AI browsers. Someone executes a prompt injection via an email, instructing your payment system to transfer a billion dollars to a hacker. The system checks to see if there's a billion dollars in the account. Whether that full amount is there or not, it figures out a way to sweep the maximum available balance and sends it directly to the hacker's account.
What happens next? You're out of business. The transaction happens in the middle of the night, and you don't even know it's taking place. This is a worst-case type of scenario for prompt injection.
If you were to advise a small business or an up-and-coming startup who might not have a big cybersecurity team, what are some simple steps that they can take to reduce these dangers drastically?
Don't allow AI to make decisions. This will safeguard you most of the time. Use AI to increase your productivity, but at the same time, keep a human in the loop to make the final decisions.
Basically, AI is a probabilistic system, not a deterministic system. In a deterministic system, every possible use case is discussed, designed, and then implemented. In a probabilistic system, the model learns from data and then predicts the most appropriate action, but you can never guarantee the outcome with absolute certainty. That is always going to be a problem.
We can utilize AI for everything else, like drafting a document or cross-checking information. But when it comes to decision-making—like whether to pay a person a certain amount right now, how much to pay them, or verifying if they are even the right person to pay—these are critical decision points that require questions, logical reasoning, and context. All of these points must be vetted by human beings. Humans have the complete picture of what is going on and what needs to be done, so they are the ones who can make the right call. Whether it is for a small business or a large enterprise, the final decisions must always be made by a human.
How do you suggest security operations teams change their approach towards AI to circumvent the dangers of prompt injection?
Have people kept at vantage points to verify things that are being done by AI. For instance, you may use AI to triage and classify all the alerts as false positives, true positives, and incidents. But you must always double-check the false positives classified by AI to see whether all these errors are really false positive or not. If you completely rely on AI to do the triaging, classifying one true positive alert as a false positive might have a major impact on your business.
But AI can bring in a real increase in productivity for security operations teams. There's a psychological phenomenon termed alert fatigue, where people monitoring large volumes of security alerts get fatigued because they are constantly bombarded by false positives. Now, since AI does the initial filtering of false positives, humans can concentrate better on the actual alerts. As they're no longer exposed to alert fatigue, the quality of work that they deliver can actually increase.
In your 25 years of professional experience, you've probably seen all kinds of cybersecurity threats—viruses, ransomware, data breaches. In that line, do you see prompt injection as a completely new kind of an attack or is it not all that different from the previous kinds of attacks? What does this spell for the future of cybersecurity?
Injection attacks are actually an old technique—SQL injections, command injections, and XSS attacks have been happening for a very long time. Now, prompt injection is the same kind of threat, but working from a different dimension.
AI browsers operate on the premise that convenience is the primary motivation for the end user. But privacy and security are exactly opposite to this. When we prioritize security, convenience takes a back seat. Now, we have to rethink all our defense mechanisms. Because of prompt injection, we can no longer rely on user behavior—we never know whether an AI is executing a particular task from the user's machine or if it's the actual user. So we need to rethink whether we need to monitor all activities from a machine or if we have to follow a wait-and-watch approach.
In my experience, security is always evolving; it's never a solved problem. There will be a problem, you go and solve it, and then the attacker finds a way to evade your defense. Then you go solve that thing, and they will again find a way to evade it. It's like a whack-a-mole problem: You go and shut a particular place, and some other place will be open. You go and close it, and another place will open. This is just how the system is, and AI is a new variable brought into the system. This will also be solved, but it will take a different dimension. Again, that needs to be solved, and it will take another dimension, and again, that will need to be solved. It's a never-ending problem.
Ulaa offers Ulaa Enterprise, a security-first browser built specifically for businesses to give IT teams visibility and control over all browser-related activities, ensuring real-time threat prevention, data protection, and user-level enforcement.
