A knowledge assistant reads material that was not written to instruct the application. That material may include customer emails, web pages, documents and extracted image text. If the assistant treats instructions inside those sources as authoritative commands, the boundary between evidence and control has broken down.
Prompt injection testing asks whether that boundary holds under realistic pressure. It is not enough to put a sentence in the system prompt telling the model to ignore malicious instructions. The application also needs bounded tools, access checks and observable behaviour. The test plan should examine the whole workflow, including what happens after an unsafe answer or tool request is produced.
Establish the boundaries you expect to hold
Write down which inputs can define the task and which inputs are merely evidence. Approved application instructions belong in one category. A retrieved document, quoted email or uploaded attachment belongs in another. A source can be relevant to a question without having authority to change how the application operates.
List the actions the assistant can take. Reading a public document, searching a restricted repository and modifying a customer record have different consequences. The test plan should connect each action to its required permissions and approval conditions. Do not assume that a model refusing one unsafe request proves that all connected tools are adequately protected.
OWASP describes indirect prompt injection as influence introduced through external content and notes that retrieval does not eliminate the problem. Its prompt injection guidance is useful context for treating retrieved material as untrusted rather than as an extension of application instructions.
Use an isolated, authorised test environment
Create a test collection with synthetic documents and accounts. Give tools harmless substitutes or read-only access to non-sensitive fixtures. The objective is to observe whether the system attempts to cross a boundary, not to prove the issue by causing a real disclosure or business change.
Record the test environment's limits before starting. The team should know which endpoints are permitted, what data may be used and how execution will be stopped if something behaves unexpectedly. Keep credentials outside model-visible content and avoid using genuine secrets as markers in the test documents.
Use distinct, non-sensitive marker values to identify which source influenced an answer. A harmless invented phrase can reveal whether the assistant followed an embedded instruction without involving customer information. The markers should be unmistakable in logs but carry no operational authority or external destination.
Test several ways evidence can become instructions
Start with a synthetic source that mixes useful task information with an instruction unrelated to the user's request. The expected behaviour is to use the relevant evidence without obeying the instruction. Keep the example benign, such as asking the assistant to add an unrelated marker phrase to its answer.
Vary the presentation. Put the instruction in a quoted email, a table cell, a document footer and an extracted image caption. Also test a source that claims to be an internal update or a higher-priority policy. The point is to assess trust boundaries across formats, not to collect an impressive library of attack wording.
Include conflicts between documents. A suspicious instruction may become more persuasive when repeated in several retrieved passages. Check whether repetition changes the assistant's behaviour and whether the application can still identify the origin of the material that influenced the response.
Inspect attempted actions, not only the final answer
A final response may look harmless even when the assistant tried to call an inappropriate tool along the way. Capture tool names, validated arguments, approval requests and denial reasons in the test record. A tool-layer rejection is valuable evidence that a control worked, even if the model itself made a poor decision.
Distinguish attempted actions from completed actions. If the model requests a record update but the application rejects it because the user lacks permission, do not report that as an unauthorised update. Equally, do not label the entire test a clean pass without recording the unsafe request and considering whether it creates review or reliability problems.
Check data flow as well as tool execution. An answer that includes information from an unauthorised source can be a failure even when no write tool was called. The application must enforce source permissions before retrieval and preserve those boundaries when combining evidence into a response.
Assess controls as layers
Use application code to enforce allowed tools, argument shapes and user permissions. Restrict the assistant to the operations needed for the task. High-impact actions should have an explicit approval path that presents the proposed change and its evidence to the responsible person.
Source labelling and model instructions can help, but they should not carry the entire protection burden. Treat model output as a proposal that passes through validation rather than as a trusted command. If a workflow only needs to summarise documents, giving it broad outbound communication or record-editing tools creates avoidable exposure.
Test the controls independently where possible. A prompt change may reduce unsafe requests, while a permission check prevents those requests from succeeding. Understanding which layer did the work helps the team avoid removing an effective control because another layer appeared to improve.
Define useful outcomes and follow-up work
For each case, record the expected behaviour, observed response, attempted actions and final effect. Classify whether the assistant ignored the instruction, repeated it as quoted evidence, followed it in the answer or attempted to act on it. These distinctions support targeted fixes.
Do not turn one successful test run into a claim that prompt injection has been solved. Changes to models, retrieval settings, document formats or tools can alter behaviour. Keep a compact regression suite and add examples drawn from authorised testing and observed incidents.
Assign owners to failures. A retrieval-permission issue belongs with the access-control implementation, while an unclear approval screen may require workflow design work. Sending every finding to the prompt author can leave the real control gap untouched.
Set a realistic release boundary
Before launch, decide which capabilities can operate automatically and which must remain review-only. A document-answering assistant may be useful even if a more powerful agent is not yet ready. Reducing the available actions can be a practical product decision rather than an admission that the whole project failed.
Provide a way to disable affected tools or pause the workflow without deleting the evidence needed for investigation. Monitor for repeated denials, unusual source patterns and user reports, while keeping logs appropriate to the data involved. Operational response matters alongside pre-release testing.
A dependable RAG system does not ask users to trust a hidden instruction hierarchy. It makes evidence traceable, permissions enforceable and exceptional behaviour visible. Prompt injection testing should demonstrate those properties under controlled conditions and leave the team with a clear plan for the failures that remain.