|
As rogue AI agents cause security problems and product outages even at sophisticated companies such as Meta and Amazon, major firms such as ServiceNow as well as startups are developing new AI to monitor and stop them. The new AI, also known as guardian AI agents, comes in the form of a cloud application and can be laborious to set up. To use a guardian AI app, customers need to connect it to other AI agents they use—including ones developed with OpenClaw, Claude Code and Agentforce—using standard application programming interfaces or model context protocol servers that enable such connections or monitoring. Customers must then tell the guardian app how the various AI agents they use should behave, given the agents’ tendency to stray from their original mission. For instance, a company might tell the guardian AI app to make sure that whenever an AI agent generates internal financial reports, it should always check Bloomberg when the reports reference stock prices. If the report-making AI agent goes against that or another one of the company’s guidelines, the guardian agent can send an alert to employees to either stop the report-making agent or change the agent’s behavior the next time the same thing happens. “You can‘t have humans actually supervising [AI agents’] work because human brains don’t work fast enough,” said Tatyana Mamut, a former executive of Amazon Web Services and Salesforce who now runs Wayfound, a guardian AI developer. Ironically, the guardian apps are often powered by the same AI models behind products like Claude Code. That raises the question of whether a guardian AI app powered by Anthropic models would be good at monitoring agents built with coding tools like Claude Code that are also powered by Anthropic models. The rise of nascent guardian AI apps comes as numerous software and AI firms are vying to sell tools to manage the growing suite of AI agents. (See a handy table on them here.) Some software and cybersecurity firms, such as IBM and Palo Alto Networks, have also developed AI for monitoring the activities of AI agents customers use across various apps. The monitoring software aims to detect possible problems, such as when employees share proprietary data with external AI providers or chatbots. But those companies don’t appear to sell guardian or supervisor agents capable of acting autonomously. ServiceNow, which sells AI agents for automating tasks such as processing insurance claims, also sells guardian agents to monitor its own AI agents as well as other agents powered by rival firms such as Microsoft and Amazon, the company said. ServiceNow’s guardian agents are part of its AI Control Tower, which charges customers a subscription fee as well as based on their usage of it. Meanwhile, Salesforce, which sells AI agents for automating tasks such as updating data within a company’s customer relationship management system, is also considering developing guardian AI to monitor such agents, according to a person who spoke to a senior executive at the company this year. It isn’t clear if the proposed Salesforce product would monitor agents across any apps a customer uses, including non-Salesforce apps. Unilever’s Prerogatives But Sam Dover, former head of AI strategy at Unilever, said companies that sell AI agents may not always be incentivized to develop guardian-type AI whose job is to poke holes into how the agents are performing. “One of the prerogatives at Unilever [was] wanting that…independent vendor of AI governance,” he said. To that end, Dover said Unilever has been a customer of Holistic AI, a six-year-old firm that develops software to monitor internal corporate AI usage and the performance of customer-facing AI apps, similar to ServiceNow’s Control Tower. Holistic has launched some guardian agents in preview and plans to officially launch them later this year, co-founder Adriano Koshiyama said. Palo Alto, Calif.-based CredoAI, which monitors the performance of customers’ AI applications or models, said it has also launched guardian agents in a private preview to a limited number of customers with an undisclosed pricing model. Wayfound, based in San Francisco, has about a dozen paying customers, primarily in the financial services and tech industries, Mamut said. Hedge funds use its guardian agents to monitor the AI agents they use to create research reports, she said. The startup charges customers on a subscription basis with extra fees based on the amount of work completed by the AI agents that Wayfound’s product monitors. The startup sells a $750-per-month subscription covering 10,000 tasks completed by agents that its software monitors, for instance. Salesforce last year included Wayfound as a “monitoring partner” to help Salesforce customers oversee or supervise the performance of its agents developed with Salesforce’s Agentforce tools. (Wayfound has four full-time employees and raised about $3.2 million in capital in late 2024.) Another guardian AI developer, Avon AI, has a similar business model. The Israeli startup, which was founded last year, charges a licensing fee to customers and adds an additional cost for every 100,000 AI agent conversations its guardian agents monitor, co-founder Amit Segev said. Segev said the company has multiple paying enterprise customers who have signed multi-year contracts but wouldn’t disclose pricing details.
| 
