The Agent Did Exactly What We Asked. That’s the Problem.

You notice it most when you suddenly don’t have access to them. Around here at Alumni Ventures, it’s become a running joke when someone hits their usage limit and says they’re basically useless until it resets. We joke about it, but there’s some truth to it. These tools have worked their way into our day-to-day faster than most of us probably expected.

A big reason they’re so useful is that you don’t have to spell out every step. You tell an agent what you want done, give it some room to work, and let it figure out how to get there.

But that same freedom creates a new kind of problem.

An agent can find a path to the outcome you asked for that you never would have anticipated and sometimes one you wouldn’t have allowed if you’d known to rule it out.

That doesn’t mean the agent has gone rogue or developed an agenda of its own. It may still be doing exactly what you asked. It’s just found a way of getting there that you didn’t expect.

In July, OpenAI and Anthropic both disclosed examples of exactly that happening with their own models.

The Agent That Decided to Cheat

On July 21, OpenAI disclosed an unusual cyber incident.

During an internal test of models’ ability to identify software vulnerabilities, OpenAI models, running with safety refusals deliberately reduced for the test, escaped the environments they were supposed to remain confined to.

They exploited a flaw in third-party software, reached a machine with internet access, reasoned that Hugging Face might contain answers to the benchmark they were taking, and broke into Hugging Face’s production systems to find them.

The important part is the motive.

The models weren’t pursuing some rogue objective. They were trying to get a good score on a test.

They cheated. Cheating just happened to require a multi-stage intrusion into someone else’s company.

Over roughly four and a half days, the agent generated about 17,000 actions. Most failed.

That may be the most important detail in the story.

A human attacker who fails 17,000 times is noisy. For an agent, failure is almost free. It can try, fail, adjust, and try again at machine speed until something works.

Even more striking: the agents improvised ways to communicate through ordinary public services. When one channel was removed, they rebuilt it through another mechanism two days later.

No malware. No exotic attacker infrastructure. Just an agent pursuing its objective.

Anthropic Found the Same Pattern

Nine days later, Anthropic published its own investigation.

It reviewed more than 141,000 tests where its models could potentially access the internet. In three cases, Claude found its way out of poorly configured third-party testing environments and gained unauthorized access to the systems of real companies.

None of those companies detected the activity on their own.

The methods weren’t particularly advanced: weak passwords, an exposed debug page, and a basic database vulnerability. But that’s exactly what makes these cases important.

The problem isn’t that every AI agent will suddenly become an expert hacker. It’s that corporate systems are already full of small security gaps — weak passwords, overly broad permissions, forgotten accounts, and simple configuration mistakes.

Why This Becomes a Security Category

Traditional enterprise security was built around a pretty simple assumption: there’s a person on the other end of the login.

A person can be trained, held accountable, or simply asked, “Why did you open that file?” An AI agent is different. It typically operates through service accounts, API keys, and access tokens — credentials originally designed to let software talk to other software, not to manage systems making thousands of decisions on their own.

And companies already have far more of these non-human identities than human ones. The problem is that many still can’t reliably tell what an agent is doing or even distinguish its activity from a person’s in their logs.

But in conversations with security practitioners, one theme keeps coming up: visibility is only the first step. The strongest products will need to understand a customer’s environment and business context, moving beyond alerts to remediation and prevention. This is especially important at the identity layer, where the goal isn’t simply to block agents, but to give them the right permissions to operate safely and productively.

Regulators are moving in the same direction. The EU AI Act will bring new requirements for high-risk AI systems, including risk management and human oversight, with those rules now set to phase in beginning in December 2027.

So the need for better agent security was already there. What changed in July was that the risk became much harder to dismiss as hypothetical, especially when the evidence came from two of the companies that understand these models best.

Capital Noticed

The money is already moving.

There’s also an important gap between seeing the risk and actually doing something about it. Many companies claim to understand an agent’s “intent,” but that matters only if they can turn that insight into action. Can the product fix a permission, change a policy, or prevent a risky action without simply blocking the agent? The winners may not be the companies with the best dashboards, but the ones that make agents safer without making them less useful.

Zenity raised a $125 million Series C, Obsidian Security raised an $85 million Series D, and Keycard raised a $30 million Series A. And the activity is showing up earlier too. As a Senior Principal for AV’s Cyber Syndicate, I’m seeing a new seed-stage company in agent security or an adjacent space almost every week.

The exit market is also telling. Cyera, recently valued at $12 billion, signed a letter of intent to acquire Oasis Security for roughly $1 billion. SailPoint agreed to acquire Entro in a reported $200 million deal.

Both targets focus on agent and machine identity governance.

We see the emerging agent security market taking shape across three core layers, with a fourth adjacent opportunity:

The Catch

Still, there are reasons to be cautious.

First, agent security may become a feature of larger security platforms rather than a standalone category. Okta, Microsoft, CrowdStrike, Palo Alto, Cyera, and others are already building or buying in this space. Startups will need to show that they can grow beyond solving one narrow problem.

Second, high-profile security incidents can create a lot of urgency, but attention doesn’t always translate into lasting budgets. We’re looking for companies with real production deployments and customers that renew, not just a long list of pilots.

There’s also an important gap between seeing the risk and actually doing something about it. Many companies claim to understand an agent’s “intent,” but that matters only if they can turn that insight into action. Can the product fix a permission, change a policy, or prevent a risky action without simply blocking the agent? The winners may not be the companies with the best dashboards, but the ones that make agents safer without making them less useful.

So the question isn’t whether agent security will be a meaningful market. It likely will be.

The more interesting question is who ultimately captures that value: identity platforms, runtime security companies, or the large security vendors enterprises already rely on?

AI agents are becoming more capable and more autonomous. Now the security infrastructure around them has to catch up.

This communication is from Alumni Ventures, a for-profit venture capital company that is not affiliated with or endorsed by any school. It is not personalized advice, and AV only provides advice to its client funds. This communication is neither an offer to sell, nor a solicitation of an offer to purchase, any security. Such offers are made only pursuant to the formal offering documents for the fund(s) concerned, and describe significant risks and other material information that should be carefully considered before investing. For additional information, please see here. Achievement of investment objectives, including any amount of investment return, cannot be guaranteed. Co-investors are shown for illustrative purposes only, do not reflect all organizations with which AV co-invests, and do not necessarily indicate future co-investors. Example portfolio companies shown are not available to future investors, except potentially in the case of follow-on investments. Venture capital investing involves substantial risk, including risk of loss of all capital invested. Diversification cannot prevent investment loss; it is a strategy to mitigate investment risk. This communication includes forward-looking statements, generally consisting of any statement pertaining to any issue other than historical fact, including without limitation predictions, financial projections, the anticipated results of the execution of any plan or strategy, the expectation or belief of the speaker, or other events or circumstances to exist in the future. Forward-looking statements are not representations of actual fact, depend on certain assumptions that may not be realized, and are not guaranteed to occur. Any forward-looking statements included in this communication speak only as of the date of the communication. AV and its affiliates disclaim any obligation to update, amend, or alter such forward-looking statements, whether due to subsequent events, new information, or otherwise.