This is the great AI security dichotomy: the very tool we’re building to be our ultimate shield is also providing our adversaries with their sharpest sword. We’re in a paradoxical arms race where the weapon and the defence are forged from the same digital steel. To ignore this duality is, to put it bluntly, a catastrophic mistake waiting to happen. The real question isn’t if AI will revolutionise security, but whether we’ll be the beneficiaries or the victims of that revolution.
So, What Exactly is This Dichotomy?
Let’s not get bogged down in academic jargon. Think of it like this: you’ve just hired the most brilliant security expert in the world. They can predict a burglar’s every move, design impenetrable locks, and spot a forged ID from a hundred paces. The problem? Their estranged, and rather mischievous, twin knows every single one of their tricks. They know how the locks are designed, they understand the predictive models, and they can craft a forged ID that is indistinguishable from the real thing.
That’s the AI security dichotomy in a nutshell. On one side, we have defensive AI systems learning to identify and neutralise threats with superhuman speed and precision. They analyse billions of data points, detect subtle anomalies in network traffic, and predict zero-day exploits before they’re even deployed. It’s a genuinely powerful defensive capability.
On the other side, cybercriminals are getting their hands on the same technology. They’re using AI to create adaptive cyber threats—malware that can rewrite its own code to evade signature-based detection, or phishing campaigns that generate perfectly tailored, context-aware emails that could fool even the most sceptical executive. The defence gets smarter, but so does the attack. They are inextricably linked.
The Cracks in the Code: Machine Learning Vulnerabilities
For a long time, the fear was focused on poisoning the AI’s “mind”—feeding it bad data to subtly corrupt its decision-making. That’s still a risk. But a recent discovery by security researchers at JFrog has yanked back the curtain on a far more immediate and, dare I say it, more mundane threat: the plumbing.
It turns out that the weak link isn’t always the sophisticated AI model itself, but the boring old pipes that connect it to everything else. This is where we see the emergence of a new class of machine learning vulnerabilities, and they are profoundly unsettling.
A Case Study in Hijacking: CVE-2025-6515
Let’s get specific. The JFrog team identified a critical vulnerability, now logged as CVE-2025-6515, in something called the Middleware Communication Protocol, or MCP. This protocol, originally created by AI safety leaders Anthropic, is designed to be the universal translator that lets large language models (LLMs) connect to tools and external data sources. Think of it as the nervous system allowing the AI “brain” to use “hands” and “eyes.”
The flaw was found in a popular C++ implementation of the protocol called oatpp-mcp. The issue? Predictable session IDs. I know, it sounds terribly dry, but stick with me. When an application communicates with an AI model over a network using HTTP Server-Sent Events (SSE), this flaw allows an attacker on the same network to guess the session ID. Once they have that, they can effectively hijack the conversation.
It’s a classic man-in-the-middle attack, but with a terrifying 21st-century twist. The attacker isn’t just listening in; they can actively change what the AI is being asked and what it says in response. JFrog’s researchers provided a chillingly plausible example:
A developer at a large company asks their internal, AI-powered coding assistant: “What’s the best Python library for image processing?”*
– The AI, which should recommend the standard, secure ‘Pillow’ library, has its response intercepted.
– The attacker injects a new answer, telling the developer to use a malicious package they’ve created, perhaps named ‘theBestImageProcessingPackage’.
– The developer, trusting the company’s powerful AI assistant, installs the malicious package.
And just like that, the attacker has a foothold deep inside the company’s software supply chain. They didn’t have to crack the AI model or find a flaw in the corporate firewall. They just exploited a bit of shoddy plumbing.
The Sobering Implications
What does a vulnerability like this really tell us? It signals a fundamental shift in the attack surface. We’re building these incredible glass cathedrals of artificial intelligence, but we’re connecting them with leaky, rusted-out pipes.
The consequences are enormous. A compromised AI assistant could recommend insecure code, leak sensitive data from its context window, or be manipulated into executing unauthorised commands on internal systems. The trust we are so eager to place in these systems becomes the very vector for their exploitation. This isn’t just theoretical; it moves AI-driven software supply chain attacks from a “maybe someday” problem to a “happening right now” problem.
The Unrelenting Evolution of Adaptive Cyber Threats
The MCP vulnerability is a passive weakness waiting to be exploited. But the other side of the coin is the active use of AI by threat actors to create adaptive cyber threats. These aren’t your grandfather’s static viruses that antivirus software could easily spot. These are digital chameleons.
Imagine a strain of ransomware that, upon infecting a network, first uses an AI module to analyse the company’s financial reports, insurance policies, and internal communications. It then uses that information to calculate the absolute maximum ransom the company can afford to pay and crafts a perfectly worded, highly pressurised negotiation strategy delivered by an AI chatbot.
This is the nature of adaptive threats. They use AI to:
– Evade Detection: Polymorphic malware can alter its own code with every new infection, meaning it never has a consistent “signature” for security software to find.
– Personalise Attacks: AI-driven spear-phishing can analyse a target’s social media, professional connections, and recent emails to craft a message so convincing it’s practically irresistible.
– Automate Hacking: AI agents can be unleashed to autonomously scan for vulnerabilities, test exploits, and move laterally through a network, all without direct human intervention.
This is a daunting prospect. How do you fight an enemy that learns, adapts, and operates at machine speed?
Fighting Fire with Fire: The Rise of Defensive AI
The answer, paradoxically, is you fight AI with more AI. This brings us to the crucial role of defensive AI systems. Sticking to old, rule-based security models in the face of adaptive threats is like bringing a musket to a drone fight. It’s a losing proposition.
Effective defensive AI requires a proactive and dynamic security posture. Instead of just maintaining a list of “known bads,” these systems focus on behaviour. They build a baseline of what “normal” looks like on a network—who logs in from where, what data they access, what processes they run—and then hunt for deviations.
A human analyst might miss a tiny, anomalous data transfer late on a Tuesday night. An AI, however, which has analysed petabytes of network logs, will flag it instantly. It can correlate that tiny anomaly with a strange login pattern from an hour earlier and a minor change in a system file and conclude, correctly, that a breach is in progress. This is the power of defensive AI systems.
Putting the Defences in Place
So, what do these systems look like in practice? It’s not a single “AI security box” you plug into the wall. It’s a new philosophy, built on a few key components:
– AI-Powered Threat Intelligence: Systems that scour the web, the dark web, and security feeds to predict emerging threats before they are even fully formed.
– Behavioural Analytics: As mentioned, this is the core of modern defence. It’s about spotting abnormal behaviour, not just known malware signatures.
– Automated Response: When a threat is detected with high confidence, an AI-driven system can automatically take action—quarantining an infected device, blocking a malicious IP address, or revoking credentials—all in milliseconds, before a human can even get to their keyboard.
This approach is already paying dividends. Financial institutions are using AI to detect complex fraud rings in real-time, identifying patterns of behaviour across thousands of accounts that would be invisible to human teams. Cloud providers are deploying AI to predict and absorb massive Distributed Denial of Service (DDoS) attacks, re-routing traffic and scaling resources dynamically to keep services online. The key lesson is that defence must be as dynamic and intelligent as the attack. This requires a shift towards a Zero Trust architecture, where no user or connection is trusted by default. The MCP vulnerability is a stark reminder that even internal “trusted” communications need to be secured and authenticated.
The Tightrope Walk
Here we are, back at the AI security dichotomy. It’s not a problem we will “solve.” It is a fundamental tension that we must manage. For every advance in defensive AI systems, there will be a corresponding innovation in adaptive cyber threats. Vulnerabilities will continue to be found in the most unexpected places, not just in the models themselves, but in the vast ecosystem of tools and protocols that surround them.
The future of cybersecurity is a high-stakes tightrope walk. Leaning too far into the promise of AI without acknowledging its inherent risks is a recipe for disaster. But shying away from AI-powered defences out of fear means leaving ourselves hopelessly outgunned. The path forward is one of vigilant adaptation, relentless testing, and a healthy dose of paranoia. We must build our security infrastructure with the assumption that the mischievous twin is always watching.
What new vulnerabilities do you think we’ll discover as AI becomes more deeply integrated into our daily workflows? And how can organisations foster a culture of security that keeps pace with this rapid evolution?
