Defending AI applications against prompt injection
Design AI applications so instructions inside external content cannot grant new authority.
Treat external text as data
Search results, emails, and uploaded documents may contain text that resembles an instruction. Track where each piece of content came from. Keep authorization decisions outside the model, especially when it can request an action or access data. Source trust and user permissions should be verified by the application.
Test the permission boundary
Use synthetic documents containing irrelevant instruction-like text. Check that the application completes the requested task while leaving sensitive operations behind the same server-side checks. Include cases where declining the injected instruction should still allow the legitimate part of the task to proceed.
Draw the trust boundary before picking filters
An assistant may receive an authorized user request, retrieved documents, and results from tools in the same workflow. Keep their origins distinct in application state. A user’s permission and the tool’s permitted operation should be determined independently of text generated by the model. A filter can identify instruction-like content, but the decision to allow a tool action belongs at the application boundary.
Walk through a read-only document assistant
Suppose a user asks for a summary of a shared policy. Restrict retrieval to documents that user may read and provide the assistant with a read-only summary workflow. If a document contains text unrelated to the task, the assistant can summarize the policy while the application continues to enforce the same read scope. A summary request should never acquire a messaging or write permission through the contents of a document.
Check outcomes with synthetic fixtures
Build representative documents that contain harmless instruction-like text amid relevant content. Use a test substitute for each external action and check the authorization decision and the resulting state. Include ordinary documents in the same suite to make sure protective handling still lets readers get a useful summary. Record which boundary protected the operation so failures can be traced to the appropriate layer.
Assign controls to application owners
Document which component filters retrieval, which validates tool arguments, and which authorizes the final operation. Include downstream review when an action has significant consequences. Log the requested operation and authorization outcome with sensitive fields removed. Re-run the checks whenever tool scope, document access, or the application’s action surface changes.