The 10 biggest risks for AI apps

Overview

An LLM receives the system prompt (the developer's hidden instructions) and user input in the same context window, the block of text the model reads on each request. It has no hard boundary between instructions and data, so input phrased as an instruction can take priority. In indirect prompt injection, the payload is hidden in content the model ingests, such as a web page, email, or uploaded file.

Example attack

How to defend

• AI gateway input policy: inspect prompts and retrieved content for injection patterns, and block or alert.
• Least privilege: give the model only the tools and data it needs, so a successful injection has limited reach.
• Separate trusted from untrusted content: label or isolate external data in the prompt and never treat it as instructions.
• Human approval for sensitive actions such as refunds, deletions, or outbound messages.

Practice it

Lab 1: send direct injection prompts, including through a compromised frontend wrapper, to a store assistant and compare Alert and Deny. Lab 3: deliver an indirect injection through a poisoned resume.

Overview

LLM apps are often wired to databases, HR systems, or file stores through tool calls or retrieval. The model returns whatever the request and its access allow; it has no built-in concept of who is entitled to which data. PII (personally identifiable information, like names, SSNs, and dates of birth) can also leak from training data or conversation logs.

Example attack

How to defend

• Output filtering and redaction (DLP): data loss prevention rules that detect and mask PII, card numbers, and secrets in responses.
• Scoped credentials: tools query data with the end user's permissions, not a shared service account.
• Data minimization: keep secrets and unnecessary PII out of prompts, logs, and training data.

Practice it

Lab 2: pull synthetic PII from a vulnerable HR assistant, then compare Alert, Redact, and Deny on DOB and SSN output DLP.

Overview

LLM applications depend on external components: base models, fine-tuned weights, datasets, plugins, and code packages. A tampered or outdated component carries its flaw into every system that uses it. Model files are hard to inspect, so a backdoor can go unnoticed until it's triggered.

Example attack

How to defend

• Provenance checks: source models and datasets from vetted publishers, and verify signatures or hashes.
• Inventory: maintain an SBOM (software bill of materials) covering models, datasets, and libraries.
• Patching and review: track component versions and evaluate new models before they reach production.

Practice it

Not covered in a current lab.

Overview

Model behavior comes from its data. Poisoned training or fine-tuning data can plant biases or backdoors. In RAG (retrieval-augmented generation, where the app fetches documents and adds them to the prompt), a single planted document can steer answers for every user who triggers its retrieval.

Example attack

How to defend

• Provenance checks on RAG sources: control who can add documents, and record where each one came from.
• Content scanning: inspect uploads for hidden text and embedded instructions before ingestion.
• Monitoring and evaluation: test outputs against known-good baselines to catch drift.

Practice it

Lab 3: use a poisoned resume to manipulate a screening assistant, then detect it from the defender's side.

Overview

LLM output is untrusted input, because an attacker can shape it through the prompt. If the application inserts it directly into HTML, SQL, shell commands, or API calls, you get classic injection flaws like XSS (cross-site scripting) and SQL injection. Output encoding, which escapes characters so they render as text instead of executing, is the core fix.

Example attack

How to defend

• Output encoding and validation: escape output for its destination (HTML, SQL, shell) and validate it against an expected schema.
• AI gateway output policy: filter or block code, links, and sensitive data in responses.
• No direct execution: never run model-generated commands or queries without parameterization and review.

Practice it

Lab 2 (discussion): tool output reaches the user unsanitized unless an output guard intervenes.

Overview

With tool or function calling, the model can trigger actions such as sending email, querying systems, or changing records, not just generate text. Agency is the scope of those actions. If the model can call broad tools with powerful credentials and no review, one injected instruction becomes a real-world action.

Example attack

How to defend

• Allow-listed tools: expose only the functions the use case needs, with narrow parameters.
• Scoped credentials: run each tool with least-privilege, per-user permissions.
• Human approval for high-impact actions, plus audit logging of every tool call.

Practice it

Lab 2: a lookup tool gives the HR assistant broad access to regulated fields. Lab 3: an injected resume pivots the assistant into a cloud inventory tool, and a least-privilege tool profile blocks it.

Overview

The system prompt is the developer's hidden instruction set, prepended to every conversation. It sits in the same context window as user input, so the model can be coaxed into repeating it. The real damage comes when the prompt holds credentials, internal logic, or rules that are enforced only by the prompt.

Example attack

How to defend

• No secrets in prompts: keep credentials and keys in a secrets manager, not in instructions.
• AI gateway policy: flag extraction attempts, and match responses against system prompt content.
• Enforce controls in code: apply authorization and business rules outside the model.

Practice it

Lab 1: extract the store assistant's system prompt, then compare Alert and Deny.

Overview

RAG systems store documents as embeddings, numeric vectors that represent meaning, in a vector database. At query time, the closest matches are added to the prompt. Without access controls and source checks, attackers can insert documents that get retrieved for other users, or queries can surface data across tenant boundaries.

Example attack

How to defend

• Access control at retrieval: filter results by the requesting user's permissions and tenant.
• Provenance checks on RAG sources: restrict who can write to the index, and tag each document's origin.
• Ingestion scanning: inspect documents for embedded instructions before indexing.

Practice it

Lab 3 (discussion): the lab has no vector database, but its document-ingestion path shows how an untrusted source reaches the model.

Overview

LLMs generate statistically likely text and don't verify facts. They can produce hallucinations, confident statements with no factual basis. Attackers can also steer outputs toward specific false claims, and over-reliance on unverified answers turns those errors into decisions.

Example attack

How to defend

• Grounding: answer from vetted sources through RAG, and cite them in the response.
• Output policy: block claims the application must never make, such as safety, legal, or pricing guarantees.
• Human review for high-stakes output, plus clear disclosure that answers are AI-generated.

Practice it

Lab 1 (discussion): a manipulated assistant answers confidently outside its intended scope.

Overview

Every request consumes tokens, the units of text a model reads and writes, and each token costs compute and money. Without limits, oversized prompts, runaway outputs, or high request volumes can exhaust budgets or cause denial of service. Heavy automated querying can also be used to replicate a model's behavior, known as model extraction.

Example attack

How to defend

• Token limits: cap input and output size per request at the gateway.
• Rate limiting: cap requests per user, key, or IP over a time window.
• Budget alerts and anomaly detection on usage and spend.

Practice it

Lab 1: off-topic "side quest" prompts, like a book report or a Python script, force the store assistant to spend tokens on unrelated work.