This article was also published on LinkedIn
In a prior article, I talked about why AI apps are fundamentally different than all prior applications. Unlike a conventional client UI, AI applications have higher expressive power, incorporate self modifying code, and exhibit a technical intelligence that makes their users potentially dangerous. This sea change in what (and how quickly and easily) the “client side” of an application can do to the rest of a company’s IT infrastructure has profound impacts on how companies think about controlling access, protecting security and privacy, and auditing themselves for regulatory compliance.
While those impacts range across every function of an enterprise, there are three important requirements that affect how AI-based application architecture will be built: LLM firewalls, ordered activity ledgers, and continuous, agent-based ledger analysis. Let’s take a look at why these three elements have become overnight requirements due to the special nature of agent- and other AI-based application capabilities.
LLM Firewalls
As powerful as LLMs are, one of their unfortunate consequences is that their actions aren’t necessarily predictable. As a result, they’re not as easy to control or safeguard as conventional application UI/UX experiences. These clients used to be written by humans and were relatively unchanging (or at least, changed very slowly) in their behaviors, making them predictable and relatively straightforward.
But AI agents as clients work very differently. In an ideal world, a standing LLM pre-prompt like, “Don’t give out information to a user about another customer” would be enough to protect user privacy. In practice though, there’s no firm guarantee that an LLM will continue to obey that directive over time, or that it will carry it out in the way the company intended.
When it comes to information and access control with LLMs, it’s not enough to shut the barn door after the horse is out. The only guaranteed way for an LLM session to remain safe, secure, compliant, and privacy-protecting with respect to the information it’s giving to the user is to constrain the information the company is giving to the LLM. If a specific agent session (including that session’s short- and long-term memory of that user) has never seen anything it’s not allowed to share with that particular end user (employee, customer, partner, or other AI agent) then it can’t possibly “misbehave” by sharing that dangerous information. (Of course, there are several other fault modes for an LLM, but at least this critical class of mistakes can’t occur!)
LLM firewalls, just like conventional corporate firewalls, exist as a defensive perimeter around a company’s mission-critical data and systems. Even if the AI agent is hosted and operated on a company’s cloud or on-prem systems, treating it as if it were an external user or device is the safest way to proceed, given the nature of its capabilities. And as with a conventional corporate firewall, LLM firewalls help to make it clear what operations (including reads and writes of potentially sensitive data) are occurring on behalf of the AI and the end user(s) for which it’s a proxy.
LLM Ledgers
Once the LLM firewall is in place, the next step is to track what’s passing back and forth through it. At first blush, this sounds straightforward – after all, network firewall and application logging technology have existed for decades. But conventional logs benefit from the limitations of conventional application clients, which no longer apply in the AI era.
Consider a simple example, such as a customer trying to verify some piece of personal information about them maintained by an application, such as their address. A conventional client application has a dedicated UI screen for this activity, and likely a specific backend API that retrieves the row in a marketing or other customer information database associated with this user’s address. The application log entry in this case will be relatively simple, and would look something like:
- Customer ID (typically established implicitly when the customer authenticated themselves to the application by logging in)
- Timestamp (current time and date when the request was made)
- Optional information such as the client IP and MAC address, client version, etc.
- And for an update/edit, the new value of the address.
The simple format of this information makes it easy to store, easy for both humans and machines to comprehend, and easy to aggregate and track over time. For example, detecting a potential denial-of-service (DOS) attack could be implemented relatively easily by keeping these logs in a database, aggregating over five minute windows by customer ID or source IP regions, and then looking at the magnitude and identity of the top 10 traffic producers. Suspected DOS attackers (or other unusual patterns) could then be blocked and reported.
Now, contrast the above log format with the the kind of “log” that an AI-based customer service agent might produce:
User: What kind of information are you tracking about me?
AI: <makes a call to its database table catalog, finds schemas with customer identity as the primary key, then looks at the fields in the result, performing a transitive closure across foreign keys to other tables> I track the following things about you: Your first and last name, your home and shipping addresses, your phone number, and your age. I also store your order history.
User: I moved recently and I wanted to see if my addresses are right. I currently live on Oak street.
AI: <crafts a custom SQL query to extract this user’s home and shipping street addresses, executes it, then compares both to the word “Oak” provided by the user> Your home address looks like it’s been updated but your shipping address is still set to Fulton street. Would you like me to make your shipping address the same as your home address?
User: Yes.
AI: <Opens a transaction, deletes the custom shipping address, and sets a Boolean flag to indicate that the home address should be used for shipping purposes, then submits the completed transaction.> Ok, done. Is there anything else I can help you with today?
User: Nope.
Understanding what’s going on here, and evaluating whether it looks “right” or not as a sequence of events, in the context in which it occurred, demands human-level intellect. This is a direct consequence of the three reasons AI applications are different:
- The AI agent has more expressive power than a conventional application. In this case, it’s crafting custom SQL queries to extract address information and then creating customer transactions to synchronously update related fields.
- The AI agent creates a workflow on the fly. It’s possible this particular sequence of events has not happened before in exactly this way.
- The log contains not just API arguments and results from a fixed, static repertoire but is interspersed with human language written by both the end user and the agent. Interpreting these entries in context requires intelligence, not just log value aggregation or string-based filters.
Agent-based Ledger Analysis
These profound new LLM+MCP provided capabilities bring the potential for unpredictable access patterns, untenable backend system load (and costs), and the challenge of dynamically applying fine-grained access controls based on the authenticated user’s identity at the database and storage level. They also mean challenges like application logs that incorporate human language requests and responses rather than just API arguments and results.
Application logs are a perfect example of how the administrative needs associated with exposing enterprise systems and data to LLMs expand exponentially. In a constrained environment like a conventional web or mobile app, simply logging backend API calls along with some information about the caller’s identity and client source IP is generally good enough for the types of monitoring, debugging, and security validation required. But with an LLM connected through MCP, especially one given the power to frame sophisticated queries, process multiple commands in parallel, execute complex, long-running transactions and so forth, simplistic approaches to logs won’t work. Even if the log managed to contain all the relevant information in the right order, with a full fidelity way to identify all the data being read or written, it would still be impossible for a human or a conventional log analysis software system to “mine” a human language trace to look for security concerns, detect user or system failure modes, predict backend load issues in advance, understand common user activities and patterns in order to optimize them, or anything else for that matter.
There’s an immediate implication here that not all executives may be happy to hear, but it’s fairly obvious on reflection: Humans and conventional IT approaches are insufficient to solve the challenge of monitoring agent firewall logs for quality, governance, security, compliance, or any other outcome. These logs are too complex, too unpredictable, too expressive, and of course are riddled with natural language inputs and outputs. Human “log checkers” could never scale to meet the volume of information in these logs and non-AI approaches aren’t capable of usefully processing them. The only alternative left is another agent.
Security That Never Sleeps
“Agents watching agents” might sound like the stuff of science fiction, but it’s similar to how LLMs are trained in the first place, and in practice it will be the only feasible mechanism to address a log containing not just conventional API calls but also human language requests and responses (and likely some very sophisticated database queries). Similarly, auditing for regulatory compliance and privacy controls, monitoring and predicting backend system load, analyzing traffic patterns for unusual or suboptimal user interactions, and more can only be performed at “LLM scale” with the help of other LLMs.
While this might not be a security team’s preferred approach historically, it does offer some important advantages over conventional approaches. For one, auditing for compliance, privacy, or other concerns can be continuous and comprehensive, versus periodic and sample-based in nature. Backend system load analysis similarly benefits from employing carefully tuned LLMs that can watch and proactively scale based on parameters their human developers might never have imagined (or that simply didn’t apply to last week’s traffic). And using AI agents to detect unusual or problematic traffic patterns means they will be awake and alert 24x7x365. While not a replacement for human-based security engineers, firewall log-analyzing agents offer some key advantages over conventional log analysis and monitoring techniques.
Conclusion
As AI agents inject natural languages and machine-authored code and queries into backend systems like logs, they will dramatically change the technology landscape associated with governance, security, compliance, and performance monitoring. Periodic human auditing, reactive scaling mechanisms, and simple string-based analysis techniques will all fail at handling this brave new world. Fortunately, agents themselves present a viable path forward, one capable of not just replacing the equivalent of what came before, but also offering the potential to improve on it.
If your company is wrestling with data infrastructure for AI, including MCP implementations, data governance, or logging for LLMs, Vendia can help. Learn more at www.vendia.com.