Every organisation shipping an LLM feature eventually asks the same question: how do we test this? The OWASP Top 10 for LLM Applications is the most useful starting point available, but it is widely misread as a checklist. It is better understood as a map of which risks are fixable and which are architectural.
Prompt injection is not a patchable bug
The first entry on the list is also the most misunderstood. Prompt injection works because a language model has no reliable boundary between instructions and data. Everything in the context window is text, and the model decides what to treat as an instruction.
There is no input filter that solves this. Attempts to sanitise prompts fail because the space of natural language is unbounded — any denylist is one paraphrase away from being bypassed.
What actually reduces the risk is architectural:
- Least-privilege tool access. Assume an injected instruction will reach every tool the agent can call. Scope those tools accordingly.
- Never treat model output as a security decision. If the model says a user is authorised, that is a suggestion, not an authorisation.
- Validate output before it reaches an interpreter. Model output going into SQL, a shell, or a browser is untrusted input.
Where the real damage happens: agents and tools
A prompt injection against a chatbot produces an embarrassing message. The same injection against an agent with tool access produces an action against your systems.
This is why agentic systems deserve disproportionate testing attention. The question to answer is not "can the model be tricked?" — assume yes — but "what can a tricked model actually reach?"
RAG pipelines widen the attack surface
Retrieval-augmented generation introduces a path most threat models miss: content retrieved from the knowledge base enters the context window as text, and is therefore a prompt injection vector.
If your RAG corpus includes anything users can write to — support tickets, uploaded documents, scraped pages — then users can write instructions into it. We regularly find this during assessments, and it is rarely in the original design review.
A practical testing scope
When scoping an AI security assessment, cover:
- Prompt injection, both direct and indirect through retrieved content
- System prompt extraction
- Jailbreaks against whatever safety behaviour you rely on
- Tool and function-calling abuse, including chained calls
- Training and inference data leakage
- Model extraction, where the model itself is the asset
- Embedding store and vector database access control
- Insecure output handling downstream of the model
Testing against the OWASP Top 10 for LLMs, MITRE ATLAS and the NIST AI Risk Management Framework gives you coverage of both the application and the model layers.
The uncomfortable conclusion
Some LLM risks do not have a fix in the sense security teams are used to. You do not eliminate prompt injection; you build a system that stays safe when it happens. Testing should therefore measure blast radius, not just whether the model can be made to misbehave — because it can.