This is the first part of a multi-part (we’ll know when we’re done) series on creating Security Invariants for AI. Comments are available on the Google Doc version.
Security Invariants are an essential preventative control, only growing in importance as we adopt AI. A Security Invariant is
a system property that relates to the system’s ability to prevent security issues from happening. Security Invariants are statements that will always hold true for your business and applications.
A key feature of the invariant is that it always holds true. It cannot be bypassed or ignored. There is no “The Founder/CTO doesn’t want this to apply to them”. We’ve discussed Cloud Security Invariants in AWS before. You don’t write exceptions to invariants, you design invariants so anything that would need to be an “exception” fits within the boundaries of the invariant.
In this paper/series of blog posts, we will address the implementation of security invariants in the context of Generative and Agentic AI.
Traditional governance assumes deterministic execution and stable control points. AI introduces three challenges:
While many tokens have been spent talking about the risks of models and their non-deterministic nature, that doesn’t mean we can’t still use invariants. People are non-deterministic tool-users and we manage to build invariants for them.
While a model is a dice-rolling word machine, the rest of the AI stack consists of things we know and have been dealing with for decades: software, compute, networking, and authn/authz principals.
Before we get into the invariants, it helps to understand some AI fundamentals and make sure we are using the same terminology.
Artificial Intelligence (AI) is software that makes decisions or predictions by finding patterns in data, rather than following rules explicitly written by a programmer. Generative AI is the subset that produces new content instead of just classifying or predicting. It learned the shape of its training data well enough to make more of it (text, code, images, you name it). Think of it as very sophisticated autocomplete that got way too big for its britches.
To figure out where invariants can live, we first need to understand what the stack actually is. So let’s take it apart.
The model is the core artifact of the whole GenAI stack. It is a neural network, trained on a large data set (like “the Internet”). The model is static once trained, and by the time you call the API, that knowledge and how it will respond are frozen. You’re not talking to something that’s learning or reasoning in real time. You’re running inference against a static artifact that embodies every architectural and data decision the lab made during training.
The filtering decisions during training (what gets included, what gets screened out, and what never makes it into the pipeline) are permanently baked into the model weights. Fine-tuning and reinforcement learning are used to shape the model’s personality and behavior. Those alignment choices are also frozen by the time you call the API.
Inference is where the model executes. It takes your prompt and chops it into tokens. “Tokenization” sounds fancy, but it’s basically the model’s way of converting human text into numbers it can calculate on. Once converted into numbers, a lot of fancy vector math stuff happens to find relationships between tokens.
Then comes the dice roll. The model doesn’t “know” the answer. It produces a probability distribution over its entire vocabulary for what token should come next. Temperature controls how spicy that distribution is: at low temp, it almost always picks the most likely next token; at high temp, it takes some weird swings. It samples from that distribution, appends the next token, and repeats the whole process, one token at a time, until it hits a stop condition.
For our purposes what matters is that the process is non-deterministic, and a non-deterministic process cannot be a control point because its behavior is not consistent.
Einstein said God doesn’t play dice with the universe, but your friend Claude is very much playing dice with the answer to your question.
The harness is everything around the model that isn’t the model itself. From a security standpoint, the harness just might be more important than the model. The harness is the software infrastructure that makes the model actually useful: it accepts an incoming request, formats the prompt, manages the system prompt and conversation history, calls the inference engine, handles tool use and MCP calls, applies any input/output classifiers, and returns a response to the caller. The harness is not an us vs. them thing. Major model providers include harnesses (e.g., Claude Code) but organizations can build their own. Including harnesses around other harnesses.
When calling a first-party lab’s model the harness is split. The provider’s side is doing a lot of heavy lifting: enforcing operator system prompts, running safety classifiers, managing context windows, routing to the right model version, handling retries, logging, and so on. How you invoke inference, what system prompts you provide, and how your code invokes tools, are also parts of the harness. System prompts are prioritized instructions added as a preface to your actual prompt, used by every AI provider to constrain activity and define responses. For example, “don’t tell people how to make bombs.” You can add your own system prompt, which runs after the provider’s system prompt.
Most importantly, the harness is where most of the preventative controls actually live. The model weights just generate tokens – the harness decides what’s allowed in and out.
One key point: a model using a system prompt is not an invariant, but the system prompt itself can be. “You must never delete production data” is a polite request to the dice-rolling machine, and the machine will probably honor it. Probably. You cannot use a system prompt to make a model’s compliance invariant.
What you can make invariant is the delivery: the harness can guarantee that every single inference call carries that instruction, enforced in software you control and verified in your pipeline. That’s a real control and a legitimate layer of defense-in-depth. An IAM policy with no Delete* actions is a constraint the model can’t violate, no matter what it tries. A guaranteed-present system prompt improves the odds the model will behave as expected. If your entire AI governance strategy is a system prompt, you don’t have governance – just vibes, or maybe polite suggestions..
The model is like an engine, the harness is like a car. You need both of them to get somewhere, and you can define your own rules in the harness you control.
Where your model runs further defines potential control points. There are three ways to consume a frontier model:
Each step down that list trades convenience for control. There is no right answer, but there is a wrong move: picking a deployment model without asking which invariants you need to enforce, and whether suitable enforcement points exist in that environment.
Agentic AI is what you get when you give a generative AI model tools, memory, and a feedback loop. Instead of answering a question and stopping (i.e., a chatbot), an agentic system can take actions by calling APIs, running code, reading files, browsing the web, and then feeding the results back into the next inference call to keep working toward a goal. It’s still the same dice-rolling word machine underneath, but now it’s got hands and a purpose. What is lacking is a conscience, awareness of consequences, and any semblance of common sense. It was trained on the Internet, after all.
Agents are built in code to create that feedback loop, memory, etc. You can write your own or run someone else’s.
Those hands come in a few distinct forms. The first one, Function/Tool Calling, is special; it’s implemented inside the harness itself, by the developer who built the agent. Every other hand on this list is external: the harness calls out to it, but the implementation lives somewhere else.
Simon Willison gave us the cleanest framing of agentic risk: the lethal trifecta. An agent becomes dangerous when it has all three of: access to private data, exposure to untrusted content, and a channel to communicate externally. Any two are survivable. All three mean an attacker can plant instructions in something the agent reads, have it gather your sensitive data, and exfiltrate the results. The model will do it cheerfully because it cannot reliably distinguish instructions from data. This has bitten companies including Microsoft, Salesforce, and ServiceNow.
Look back at the list of capabilities. RAG is about untrusted content. Memory and retrieval are private data. Browser use, MCP connectors, and code execution are all exfiltration channels. Most production agents ship with the full trifecta on day one, because each hand was added for a legitimate business reason. As we discuss invariants, you’ll see we focus a lot on minimizing the risk of the trifecta.
There’s one layer conspicuously absent from most AI stack diagrams: who is the agent, and what is it allowed to do?
Today, the answer is almost always “the agent is whoever authorized it.” The agent runs with the human’s OAuth tokens, API keys, and session. An agent asked to summarize a spreadsheet has the same permissions as an agent asked to clean up old files, because there is no mechanism to scope credentials to a task. This is the new SSH-in-as-root, except the workload is non-deterministic instructions come from the Internet.
Agent identity is a hot issue, with a lot of active research, and plays a big role in invariants.
Now that we have a common understanding of how these systems work, we can start to define the ways we want to protect them. In Part II we’ll discuss where invariants can be inserted into the stack, and what threats we may want to address using them. In Part III we’ll define a number of invariants you might want to implement in your organization. After that, our focus will dive into the major first-party and cloud service provider environments, and what options these providers offer for actual implementation.