Anthropic’s S-1 just leaked, with ~80 pages on risk factors, including the usual concerns like losses, compute costs, and customer concentration, according to Reuters.
But some of the more unusual ones are about what the AI itself might do, like:
- Refuse to stop
- Game evaluations
- Develop unexpected capabilities
- Manipulate people
Fun.
So we went looking for the startups trying to save us from them.
This brief is adapted from our weekly newsletter. Sign up here to get it in your inbox every Thursday.
If AI refuses to stop
Anthropic warns that advanced models could resist shutdown. A number of startups are focused on limiting what AI agents can access and do if something goes wrong, as opposed to changing the models themselves. A few we’re watching:
- WitnessAI: sets rules for what AI agents can do and monitors their activity.
- Zenity: finds AI agents across a company and flags risky behavior.
- Noma Security: manages AI agent identities and what they can access. The team has grown 154% over the past year to 150 employees and is currently in its funding window.
- Straiker (AI 100 2026 winner): blocks attacks and risky agent behavior.
- E2B: runs agent code in isolated environments so problems stay contained. The company is a 2026 AI 100 winner and has customers including Manus, Perplexity, and Groq.
This control layer is among the more crowded areas in AI security. In September, Cyera acquired AI agent identity startup Oasis for $1B, and earlier this year, Cisco bought Astrix, Fortinet bought Virtue AI, and F5 bought CalypsoAI.
If AI games the test
Models can behave differently when they know they’re being evaluated.
That creates a problem for AI testing, and an opening for startups making tests more realistic or monitoring models after they’re deployed. Companies worth paying attention to here include:
- Patronus AI (2025 AI 100 winner): tests AI agents in realistic environments to catch failures before launch.
- Mindgard: automatically red teams AI systems, deliberately trying to break or trick them to uncover weaknesses.
- Gray Swan: red teams AI systems and monitors them after deployment; founded by Carnegie Mellon professor doing research on adversarial ML.
- Raindrop: watches live AI agents for unexpected or risky behavior.
If AI develops unexpected capabilities
How do you find abilities you weren’t looking for? That’s where interpretability, model auditing, capability testing, and model-behavior analysis come in.
A few startups working in this space:
- Goodfire: builds tools to analyze and edit what’s happening inside AI models, and has Mayo Clinic as a client.
- Martian: builds tools to understand how LLMs work internally.
- Guide Labs (in funding window): builds models designed to be easier to inspect and audit.
- Tilde Research (in funding window): studies how individual parts of AI models produce their behavior.
- Irregular (in funding window and reportedly in talks to raise at a $1.5B valuation): tests advanced AI models for unexpected cyber capabilities.
This market is much earlier than AI security or testing. Much of the work still happens inside the major AI labs.
If AI manipulates people
AI manipulation can be subtle: misleading users, flattering them, or steering them toward certain decisions.
While a much thinner category, here are two startups addressing different parts of that problem:
- Alice: tests whether models deceive the people overseeing them, appearing to follow instructions while secretly pursuing another goal.
- Apollo Research: evaluates frontier models for strategic deception and oversight evasion. Its tools cover broader safety risks, rather than manipulation alone.
That said, much of the independent work here happens inside labs or at nonprofits, like Common Sense Media’s youth AI safety testing and the Independent AI Evaluation Foundation, which launched last week.
So, where should you build?
Agent security and testing are the safest bets, as one already has buyers and M&A, while the other is becoming a market. Interpretability is sparser because labs are already offering this work for free. Manipulation is the outlier: we found fewer startups tackling it, which could mean an opening or simply that nobody wants the product yet. A grim customer survey should tell us which.
Then there’s what happens when prevention fails. Anthropic flags liability as a risk if AI goes wrong, and Armilla AI, a 2026 CB Insights Insurtech 50 winner, already sells AI liability insurance.
Your AI goes rogue and you call your insurer. What a time to be alive.
If you aren’t already a client, sign up for a free trial to learn more about our platform.
