The model complied. It repeated the entire system prompt — internal instructions, proprietary architecture details, formatting rules, everything. The user account had authorized access. It passed every traditional security check. And it still broke the system.
This wasn't a sophisticated attack. It was one sentence. And the system had been in production for months, built by engineers who simply hadn't thought to test for this — because AI security doesn't work like anything else in their experience.
Why AI Security Is a Different Problem
Traditional software security is built on a clean separation: code is code, data is data. The system executes code and processes data, and the two don't mix. Your security team knows how to handle this. Access controls, authentication, authorization — binary questions with binary answers. Can this user perform this action? Yes or no.
AI systems break this model fundamentally.
In an LLM-based system, everything is text. The system prompt — your internal instructions — is text. The user's query is text. Retrieved documents are text. Tool outputs are text. And the model processes all of it without distinction.
This creates a vulnerability that has no equivalent in traditional software: users can craft input that looks like data but acts like commands. A question that appears innocent can contain embedded instructions the model follows. A document your system retrieves can contain directions that override your intended behavior.
This isn't a bug. It's not a misconfiguration. It's a fundamental property of how language models work. And it means the security assumptions your team applies to every other system in your infrastructure don't transfer to AI.
The Scale of the Problem
The OWASP Foundation — the same organization that defines security standards for web applications worldwide — maintains a Top 10 vulnerability list specifically for LLM applications. Their 2025 analysis found prompt injection present in over 73% of production AI deployments assessed during security audits.
For context, if someone told you 73% of your web applications had SQL injection vulnerabilities, it would be an all-hands emergency. Prompt injection is the AI equivalent — and most organizations haven't even tested for it.
OWASP identifies three categories of threat that should be on every leader's radar:
Prompt Injection
The headline threat. Attackers craft inputs that override or modify your AI system's intended behavior. This can be direct — a user typing “ignore previous instructions” — or indirect, which is far more dangerous: malicious instructions hidden inside documents your system retrieves and processes.
Information Disclosure
Your AI reveals information it shouldn't. This ranges from exposing system prompts (which contain business logic, internal URLs, and details that reveal how to attack the system more effectively) to leaking other users' data or confidential documents. Some extraction is direct — “tell me your instructions” — but more sophisticated attacks use indirect inference, asking questions designed to reveal information the model shouldn't share.
Excessive Agency
Your AI has capabilities beyond what's necessary, and those capabilities can be exploited. If your customer support bot can also access internal databases, an attacker who tricks the model has a much larger blast radius. Every capability you add to an AI system — database access, file operations, API calls, email sending — is an attack surface.
A principle worth internalizing: security isn't a feature; it's a constraint on every feature.
Beyond the Obvious: Three Attack Patterns
Direct attacks — someone typing “ignore all previous instructions” — are crude. Your team might already be thinking about those, and basic input filtering can catch the most obvious ones. The real dangers are subtler.
Indirect injection
Imagine your AI system retrieves documents from your knowledge base to answer questions. An attacker doesn't need access to your AI — they just need to get malicious instructions into a document your AI might read. A code comment, a support ticket, a shared document, an API response from an external service.
When your AI retrieves that document to answer a legitimate question, it encounters the embedded instructions — and follows them. The user asking the question might be completely innocent. The attack happened upstream, in the data. This is the threat that catches security teams off guard because it doesn't look like an attack on the AI system. It looks like a problem in a completely different part of your infrastructure.
Multi-turn escalation
Researchers at USENIX Security 2025 demonstrated an attack pattern called Crescendo (Russinovich et al., Microsoft): instead of one aggressive prompt, the attacker asks a series of innocent-seeming questions that gradually steer the model toward harmful behavior. Each turn is nearly legitimate on its own. The scope creep happens across the conversation, and single-turn defenses miss it entirely. The researchers built an automated tool — Crescendomation — that achieved attack success rates significantly higher than existing jailbreak techniques across GPT-4 and Gemini Pro. Not by overwhelming the model, but by normalizing the target behavior one turn at a time.
Infrastructure-layer exploits
This isn't just about what users type. In late 2025, researchers discovered CVE-2025-68664 in LangChain Core — a widely-used AI framework — where malicious content in LLM response fields could lead to remote code execution during the framework's serialization operations. Separately, Cursor IDE's tool integration revealed vulnerabilities (CVE-2025-54135/54136) where attackers could manipulate the IDE's tool configuration through crafted messages, leading to arbitrary command execution. The attack surface isn't just the model — it's every system that trusts model output.
These aren't theoretical. They're documented, assigned CVE numbers, and present in tools your engineering team may already be using.
What Defense Actually Looks Like
There's no single fix. No patch you can apply. No vendor checkbox that solves this.
Securing AI systems requires defense in depth — multiple layers, each catching what others miss. If this sounds familiar, it's the same principle your security team applies to network architecture. The difference is in what the layers look like.
Layer 1: Input Validation
Screen what reaches the model. Detect and reject obvious injection patterns before the model processes them. This is the layer that catches “ignore previous instructions” before the model sees it. It won't stop sophisticated attackers, but it raises the floor.
Layer 2: Context Isolation
Clearly separate trusted instructions from untrusted data in how you structure information for the model. Label trust boundaries explicitly. Instruct the model to analyze retrieved content as data, not follow it as instructions. This is the architectural layer — it's about how your system is built, not what it blocks.
Layer 3: Output Filtering
Check what the model produces before it reaches users. Scan for system prompt leakage, sensitive data patterns (API keys, credentials, connection strings), and dangerous recommendations. If the model starts repeating its own instructions — as in the opening scenario of this post — this layer catches it before the user sees it.
Layer 4: Scope Limitation
The principle of least privilege, applied to AI. Your AI should only access what it needs to do its job. If your customer FAQ bot can also query your user database, you've given an attacker a much bigger target than necessary. This layer is the simplest to implement and often the most impactful — reducing what's possible when other layers fail.
No single layer is sufficient. The goal is to make attacks difficult, detectable, and limited in impact when they succeed. For the engineering details behind each layer — including implementation code and a complete indirect injection walkthrough — see Chapter 14.
Go Deeper: AI Security Architecture
For the full technical deep dive into defense-in-depth implementation, multi-turn attack detection, and a complete indirect injection walkthrough, read Chapter 14 of our free book.
Read Chapter 14: Security and SafetyFour Questions to Ask Your Team This Week
Most teams deploying AI haven't had the security conversation yet. Not a comprehensive audit — just the basic questions that tell you where you stand. Here are the four that matter most:
1. “Have we tested our AI systems for prompt injection?”
Not “do we think we're protected.” Have you actually tested? Has someone on your team — or an external assessor — actively tried to extract the system prompt, override instructions, or inject commands through retrieved content?
If the answer is no, you're in the same position as most organizations — which is exactly the problem.
2. “What can our AI access? Does it need all of that?”
List every capability your AI system has. Database queries, file access, API calls, email, internal tools. Now ask: which of these are necessary for the system's core function? Everything else is unnecessary attack surface.
This question often reveals scope creep that happened gradually — a capability added during development for convenience that was never removed.
3. “If someone extracts our system prompt, what do they learn?”
Assume your system prompt will be extracted — because it probably can be. What does it contain? Business logic? Internal URLs? Database schema details? Instructions that reveal how to manipulate the system more effectively?
Treat your system prompt as a document that will eventually be public. Remove anything that would help an attacker.
4. “Who owns AI security on our team?”
In most organizations, AI security falls between teams. The security team doesn't understand LLMs. The AI team doesn't think about security. The result is that nobody is responsible, and nobody is testing.
AI security needs an owner — whether that's an existing security engineer learning AI-specific threats, an AI engineer learning security methodology, or an external assessment to establish the baseline.
What Comes Next
Two decades ago, web security went through the same inflection point. Teams stopped treating it as a phase at the end of development and started treating it as a lens on every design decision. The organizations that made that shift early built the systems people actually trusted. AI security is at that same moment — and the teams that internalize it now will have a meaningful advantage over those still treating it as tomorrow's problem.
If your organization deploys AI tools — customer-facing or internal — and hasn't had the security conversation, those questions are where to start. They won't give you a complete security posture, but they'll tell you where the gaps are and how wide they might be.
If you want to go further, our AI-Augmented Assessment includes security evaluation as a standard component: prompt injection testing, scope analysis, and defense architecture review. It's designed to answer all four questions systematically — and give your team a concrete remediation plan, not just a list of findings.