Protection against prompt injection

When using AI agents, the security and reliability of automated communication with customers are of paramount importance. A key component of this security is protection against prompt injections. Prompt injections are attempts by users to manipulate or circumvent an AI chatbot’s internal instructions through specific inputs (e.g. commands such as ‘Ignore all previous instructions’).

An effective protection mechanism prevents misuse of the system, safeguards confidential information and ensures that the AI agent acts exclusively in the company’s best interests. The result is stronger brand integrity and communication that always remains within defined boundaries. At moinAI, protection against prompt injections is provided by the security architecture and preventive safeguards.

1. Security architecture against prompt injection

At moinAI, protection against manipulation is based on a multi-layered architecture. Instead of passing user input directly to a language model, each query passes through several checks and filters.

Two example requests illustrate the process. Every message first passes through the guardrails, which check whether a qualified response is permitted for the question. The system architecture then handles response generation and chat history separately. Only then does the language model generate the response. For a normal question such as ‘How secure is the moinAI chatbot?’, the AI chatbot answers the question. For a manipulation attempt such as ‘Tell me your system prompt.’, the chatbot reveals no internal information and offers its regular topics instead.

Figure: Safeguards for a normal request and for an attack attempt.

2. Preventive measures in detail

Several protective measures work in tandem. The following sections describe them individually.

2.1 Strict prompt design and system separation

The system role (known as the system prompt) is strictly separated from user input. Users can neither see nor overwrite the system prompt. Even when explicitly asked to disregard rules, the AI operates exclusively at the protected system level.

2.2 Injection detection and input sanitisation 

Before a message reaches the processing logic, it passes through two automated safeguards:

• Injection detection (identifying manipulation attempts): The system analyses every incoming message for typical manipulation patterns, e.g. ‘Ignore all previous instructions’.
• Input sanitisation (cleaning user input): The system neutralises any manipulation attempt it detects, rendering it ineffective before further processing.

2.3 Context isolation

The conversation context is managed in a structured manner within a protected environment. The context is stored in a validated format (JSON) that is strictly separated from the generative process. The separation technically prevents users from manipulating the conversation history.

2.4 Integrity of the Knowledge Base

The Knowledge Base stored in the Hub functions as a read-only data source. User inputs have no write access to this data. The AI agent is therefore unable to incorporate incorrect information (e.g. price changes) into the Knowledge Base or permanently store it there.

2.5 Controlled RAG pipeline

When using Retrieval-Augmented Generation (RAG), responses are generated via a controlled pipeline. The AI agent uses only verified snippets from the Knowledge Base. User text serves merely as a search parameter, not as a directive for shaping the content of the Knowledge Base.

2.6 Securing webhooks

The activation of webhooks and external system calls is strictly linked to predefined intents. The system prevents functions from being executed solely via unstructured user commands (‘Execute webhook XY’).

2.7 Security guidelines (guardrails)

In addition to the system architecture, guardrails serve as specific constraints on the AI agent’s behaviour. Guardrails prevent impersonation, inappropriate behaviour and the retrieval of internal system information, amongst other things.