This is the first part of a multi-part (we’ll know when we’re done) series on creating Security Invariants for AI. Comments are available on the Google Doc version.
Security Invariants are an essential preventative control, only growing in importance as we adopt AI. A Security Invariant is
a system property that relates to the system’s ability to prevent security issues from happening. Security Invariants are statements that will always hold true for your business and applications.
A key feature of the invariant is that it always holds true. It cannot be bypassed or ignored. There is no “The Founder/CTO doesn’t want this to apply to them”. We’ve discussed Cloud Security Invariants in AWS before. You don’t write exceptions to invariants, you design invariants so anything that would need to be an “exception” works within the boundaries of the invariant.
In this paper/series of blog posts, we will address the implementation of security invariants in the context of Generative and Agentic AI.
Traditional governance assumes deterministic execution and stable control points. AI introduces three challenges:
While many tokens have been spent talking about the risks of models and their non-deterministic nature, that doesn’t mean we can’t still use invariants. People are non-deterministic tool-users and we manage to build invariants for them. While the model is a dice-rolling word machine, the rest of the AI Stack consists of things we know and have been dealing with for decades: software, compute, networking, and authn/authz principals.
With Cloud Invariants, we’re taking an outcome “I want my CIDR ranges to be properly managed” and converting that into a plain language policy “Only the network team can create VPCs”. From there we have to find the specific enforcement points. In AWS this would be a Service Control Policy (SCP). With the enforcement point identified we implement it with a JSON document that denies specific actions unless performed by a specific IAM Role that is assigned to the networking team.
In the cloud, there are three to four major players. They each have a handful of enforcement points. AWS has Service Control Policies, Resource Control Policies, Declaritative policies, and a few service specific settings that can be set once and changes prevented with the aforementioned SCP. GCP also has organizational policies. Azure has Azure Policy and Blueprints which both control user behavior and ensure that specific configurations are present in newly created subscriptions.
Cloud Invariants primarily exist to enforce developer behavior. Cloud democratized infrastructure away from dedicated teams into DevOps. In many cases, delivery velocity became the priority, and there was no second perspective to ensure the security and resilience was taken into account.
While Cloud Invariants were primarily about preventing misconfigurations, AI Invariants are about preventing misconfigurations and misbehaviors. Since AI is non-deterministic and may do things we didn’t think we asked for, or are directed by an adversary, we use invariants to prevent a wider range of potential outcomes. We’re protecting the organization from developers, users, agents, bad guys, regulators, and broken code.
Before we get into the invariants, it helps to understand some AI fundamentals and make sure we are using the same terminology.
Artificial Intelligence (AI) is software that makes decisions or predictions by finding patterns in data, rather than following rules explicitly written by a programmer. Generative AI is the subset that produces new content instead of just classifying or predicting. It learned the shape of its training data well enough to make more of it (text, code, images, you name it). Think of it as a very sophisticated autocomplete that got way too big for its britches.
To figure out where invariants can live, we first need to understand what the stack actually is. So let’s take it apart.
The Model consists of a set of vector relationships between tokens known as weights, running in a neural network. These are static and defined during the process of training and reinforcement, but neural networks, by design, are non-deterministic and may generate different output based on the same input. The Inference Environment is where the model executes. It takes your prompt and chops it into tokens, and then uses the model weights to predict the next token. Then comes the dice roll. The model doesn’t “know” the answer. It produces a probability distribution over its entire vocabulary for what token should come next. Temperature controls how spicy that distribution is. Inference Deployment Topologies define where your model runs. There are three primary options:
Classifiers are part of the inference environment that are often considered a security control. These are smaller light weight models (or sometimes software) that scan the prompts and responses for maliciousness or ToS violations. Classifiers are how you can’t ask an advanced model for your grandfather’s nuclear weapons recipe, or in most civilized systems, non-consensual abuse material. Most classifiers run in the inference environment as an anti-abuse measure, but because they leverage an LLM themselves, they cannot be considered an invariant.
The Harness is the software infrastructure that makes the model actually useful: it accepts the incoming request, formats the prompt, manages the system prompt and conversation history, calls the inference engine, handles tool use and MCP calls, applies any input/output classifiers, and returns the response to the caller.
For our purposes, what matters is that the inference process is non-deterministic, and a non-deterministic process cannot be a control point since the behavior won’t be consistent. The harness is just standard software, and so is deterministic. This is where we can ensure our invariants live up to their name.
Agentic AI is what you get when you give a generative AI model tools, memory, and a feedback loop. Instead of answering a question and stopping (i.e., a chatbot), an agentic system can take actions by calling APIs, running code, reading files, browsing the web, and then feeding the results back into the next inference call to keep working toward a goal.
It’s still the same dice-rolling word machine underneath, but now it’s got hands and a purpose. Those hands come in a few distinct forms, which we will discuss in future sections.
Now that we have a common understanding of how these systems work, we can start to define the ways we want to protect them. In Part II we’ll discuss where invariants can be inserted into the stack, and what threats we may want to address with invariants. In Part III we’ll define a number of invariants you might want to implement in your organization. After that, our focus will dive into the major first-party and cloud service provider environments and what options these providers offer for actual implementation.