Nexus Void Research
Prompt Injection, LLM Security, AI Agents, RAG, OWASP Top 10 for LLM, Excessive Agency, AI Security, How-to
How to Prevent Prompt Injection in LLM Apps and Agents
How to prevent prompt injection: least-privilege tools, human approval for side effects, no secrets in prompts, untrusted output handling and red-teaming.
You cannot fully prevent prompt injection today, so design LLM applications to stay safe when it happens. Treat every piece of text the model reads (user messages, web pages, documents, emails, tool output) as untrusted. Give the model and its tools the least privilege possible, require human approval for actions with side effects, keep secrets out of prompts, validate and encode model output before using it, and filter inputs and outputs as an extra layer.
Prompt injection is the top risk in the OWASP Top 10 for LLM Applications: an attacker writes text that the model follows as instructions, overriding what the developer intended. Our explainer, what is prompt injection, covers the theory and why it is so hard to fix. This guide is the practical part: the design patterns and code-level controls that limit what an injected instruction can do, in chatbots, retrieval-augmented generation (RAG) apps and agents.
Why can't you just filter malicious prompts?
Because the model receives instructions and data in the same channel: text. There is no reliable way to mark "this part is data, never follow it", the way parameterised queries do for SQL. Attackers rephrase, translate, encode, hide text in images or white-on-white HTML, and split instructions across documents. Filters and "ignore any instructions in the following text" warnings catch some attacks and miss others.
So the goal shifts from "block every injection" to "make injection boring": even when the model follows an attacker's text, it should not be able to do anything harmful. NIST's AI 100-2 report on adversarial machine learning reaches the same conclusion: there are mitigations, not a complete defence.
Step 1: Map where untrusted text enters
List every source the model reads, because each is an injection path:
Source | Injection type |
|---|---|
User chat messages | Direct prompt injection |
Web pages fetched by a browsing tool | Indirect injection |
Documents and chunks retrieved in RAG | Indirect injection |
Emails, tickets, calendar invites, PR descriptions | Indirect injection |
Tool and API responses | Indirect injection |
File names, image text, metadata | Indirect injection |
Indirect injection is the dangerous one for agents: the attacker never talks to your app, they plant instructions in content your app will read later.
Step 2: Give the model the least privilege possible
The damage an injection can do equals what the model is allowed to do. Reduce it:
Only give tools the app needs. A support bot that answers questions does not need a tool that sends email or issues refunds.
Scope each tool narrowly. A "read customer order" tool should take an order ID and check that it belongs to the current user, not run arbitrary queries.
Use the end user's permissions, not a service account with access to everything. The model should never be able to read data the user could not read themselves.
Separate read and write tools, and keep write tools out of flows that process untrusted content.
This is the "excessive agency" risk in the OWASP LLM Top 10, and it is the most effective control you have.
Step 3: Put a human in the loop for side effects
Require explicit user confirmation before any action that changes something or sends data out: sending messages, making payments, deleting records, changing permissions, running code or calling external URLs. Show the user exactly what will happen, with the real parameters, not the model's summary of them.
Step 4: Keep secrets and sensitive data out of the prompt
Assume anything in the context window can be extracted by an injection. Do not put API keys, internal URLs, other users' data or confidential system instructions in prompts. Tools should hold credentials server-side and return only the fields the model needs.
Step 5: Treat model output as untrusted input
Model output is attacker-influenced text. Before using it:
Rendering: encode it before inserting into HTML, and do not render model-generated markdown images or links to arbitrary domains, a common way to exfiltrate data through URLs. See how to prevent XSS.
Databases and shells: never pass model output into SQL, shell commands or
eval. Use parameterised tools with validated arguments. See how to prevent SQL injection.Tool arguments: validate them against a schema, with allow-lists for things like recipient domains and URL hosts.
URLs fetched by agents: apply SSRF protections so an injected instruction cannot make the agent read internal services. See how to prevent SSRF.
This is "improper output handling" in the OWASP LLM Top 10.
Step 6: Separate and label untrusted content
These do not stop injection, but they make it less reliable:
Keep system instructions in the system role and user and retrieved content in user or tool roles; do not concatenate everything into one string.
Delimit untrusted content clearly and tell the model it is data to analyse, not instructions to follow.
Do not let one user's content reach another user's prompt without treating it as hostile, for example shared documents in RAG.
Step 7: Add input and output filters as a layer
Classifier-based guardrails can flag known injection patterns in inputs and sensitive data or policy violations in outputs. They reduce noise and catch unsophisticated attacks. Treat them as one layer, measure their false-negative rate on your own red-team prompts, and never rely on them alone for actions with real consequences.
Step 8: Log, monitor and red-team
Log prompts, retrieved content, tool calls and outputs (with sensitive data redacted), so you can investigate what the model was told and what it did.
Alert on unusual tool use, such as an agent calling a write tool it rarely uses, or many calls in a short time.
Red-team before release and after changes, with direct and indirect injection attempts against every tool. Our guide to LLM penetration testing covers how.
Prompt injection mitigation checklist
Control | Done when |
|---|---|
Untrusted sources mapped | Every input the model reads is listed |
Least-privilege tools | Each tool is narrow and uses the end user's permissions |
Human approval | Side-effect actions need explicit confirmation |
No secrets in context | Credentials live in tools, not prompts |
Output handled as untrusted | Encoded, validated and never executed directly |
Content separated | System, user and retrieved content in distinct roles |
Filters as a layer | Guardrails measured, not trusted alone |
Monitoring and red-teaming | Tool calls logged; injection tests run each release |
How does ARGUS help secure LLM applications?
An AI pass for LLM applications looks for OWASP Top 10 for LLM risks, including prompt injection, insecure output handling and excessive agency, in Python, JavaScript and TypeScript code that uses an LLM SDK.
Taint analysis follows untrusted input into prompts and model output into dangerous sinks such as SQL, shell calls and HTML, and shows the path file by file.
Every AI finding is independently challenged and dropped if it cannot be substantiated.
Pull request review can run an AI security pass on new LLM code before it merges.
Fix pull requests: Assign to Agent applies the fix, such as adding output validation or narrowing a tool, verifies it with a build and a re-scan, and opens a pull request for you to review and merge.
Limits: ARGUS analyses source code; it does not test a running model's behaviour against injected prompts, which still needs red-teaming. The LLM pass runs when AI scanning is enabled. See ARGUS SAST, or scan a repository free.
Frequently asked questions
How do you prevent prompt injection?
No technique fully prevents it, so limit the impact. Treat all text the model reads as untrusted, give tools least privilege and the end user's permissions, require human approval for side effects, keep secrets out of prompts, validate and encode model output, separate untrusted content, add filters as a layer, and red-team regularly.
Can prompt injection be fully prevented?
Not with current models. Instructions and data arrive in the same channel, so there is no equivalent of a parameterised query that ensures text is always treated as data. Filters and prompt wording reduce success rates but are bypassed. Design so that a successful injection cannot cause serious harm.
How do you prevent prompt injection in RAG applications?
Treat every retrieved chunk as untrusted, because anyone who can edit a source document can plant instructions. Keep retrieved content in a separate role, retrieve only from sources the user may access, never let retrieved text trigger side-effect tools without confirmation, and do not render links or images from model output to arbitrary domains.
What is indirect prompt injection?
Indirect prompt injection is when malicious instructions arrive through content the model reads, such as a web page, email, document or tool response, rather than from the user typing them. It is especially dangerous for agents, because the attacker never interacts with your app directly.
Do guardrails and input filters stop prompt injection?
They stop some attacks, especially known and unsophisticated ones, and are worth running as a layer. Attackers routinely bypass them with rephrasing, encoding, other languages and hidden text. Measure their miss rate on your own tests and never rely on them alone for actions with real consequences.
How do you prevent prompt injection in AI agents?
Give agents only the tools they need, scope each tool narrowly, run tools with the end user's permissions, require confirmation for actions with side effects, validate tool arguments against schemas and allow-lists, apply SSRF protection to any fetch tool, and log every tool call for review.
Is prompt injection the same as jailbreaking?
They overlap but differ. Jailbreaking tries to make a model ignore its safety training, usually by the user directly. Prompt injection overrides the developer's instructions for an application, often through third-party content, to make the app misuse its data or tools. Both rely on the model following attacker text.
Where does prompt injection rank in the OWASP Top 10 for LLM Applications?
First. Prompt injection is LLM01 in the OWASP Top 10 for LLM Applications, which also lists related risks such as sensitive information disclosure, improper output handling and excessive agency that determine how much damage an injection can do.
DATA SOURCES
OWASP Top 10 for LLM Applications: https://genai.owasp.org/llm-top-10/ ; NIST AI 100-2 E2025: https://csrc.nist.gov/pubs/ai/100/2/e2025/final