Challenges in Governing AI Agents

Editor’s note: This piece originally appeared on the Center for AI Safety’s website.

Leading AI companies have released a new type of AI system: autonomous agents that can plan and execute complex tasks in digital environments with limited human involvement. OpenAI’s Operator, Google’s Project Mariner, and Anthropic’s Computer Use Model all perform a similar function. They type, click, and scroll in a web browser to carry out a variety of online tasks, such as ordering groceries, making restaurant reservations, and booking flights. While the performance of these agents is currently unreliable, improvements are on the horizon. Scores on multiple benchmarks are steadily improving. The aspiration is to create AI agents that can undertake a broad range of personal and professional activities, serving as artificial personal assistants and virtual coworkers.

These developments signal a step change in the trajectory of AI technology. Unlike language models and generative AI tools that primarily produce content, AI agents can independently take actions to accomplish lengthy open-ended goals. For example, Deep Research, which was developed by OpenAI, can complete in-depth research projects by analyzing the information it retrieves, deciding to perform additional searches, and producing detailed research reports that rival those of human experts. Google’s Project Astra, meanwhile, serves as a prototype universal AI assistant that operates across multiple devices, including phones and glasses—much like Samantha in the film Her.

The economic opportunities presented by AI agents are immense, both for the companies developing them and for society at large. Applications range from automating household purchases and travel arrangements to conducting cutting-edge scientific research. Alongside these benefits, however, there are notable risks. These include malicious actors using AI agents to autonomously carry out cyberattacks and perpetrate online fraud, as well as broader concerns stemming from changes in human behavior and social structures as people increasingly delegate personal and professional tasks to AI agents. Users may even lose control over their agents, or discover that they have engaged in undesirable or unethical activity. To be clear, these risks differ from the concerns associated with ordinary content-producing language models, and stem from the distinct features of agents: their ability to take actions in pursuit of goals.

Tackling the risks posed by AI agents is difficult for several reasons. The technology is progressing quickly and being deployed in new and diverse domains, each of which presents distinct challenges. At the same time, there is limited publicly available information about AI agents. For example, until recently there were no reliable answers to the following questions:

• Which organizations are building AI agents?
• In which specific domains are they being deployed?
• What infrastructure do AI agents rely on?
• How is their performance and safety being evaluated?
• What steps are currently taken to mitigate risks from AI agents?

Ecosystem Documentation

To answer these basic questions and provide greater transparency into the development and use of AI agents, I co-led a team of researchers from MIT, Stanford, Harvard, and other institutions to create the first public database that documents the technical, safety, and policy-relevant features of deployed AI agents. The database—known as the AI Agent Index—contains 33 fields of information in relation to 67 different AI agents, all of which were manually collected through publicly available documentation and correspondence with developers of these systems. The information spans technical aspects of how AI agents are built and tested to details concerning the organizations developing them.

The study’s findings are revealing and offer important lessons for governing AI agents. We found that 75 percent of AI agents specialize in either using computers for diverse tasks (like Google’s Mariner) or assisting in software engineering, which suggests that governance efforts should focus on these broad domains rather than narrower applications of AI agents. In addition, we found that 67 percent of developers of AI agents are based in the United States (12 percent in China) and 73 percent are companies in industry, as opposed to academic institutions. Accordingly, attempts to govern the technology should, at least initially, focus primarily on the United States and should take into account the incentives of industry actors driving progress in the field.

The most striking finding of our study is that while most developers release code and documentation for AI agents, very few release information regarding safety testing and risk management. Fewer than 20 percent of developers disclosed a formal safety policy and fewer than 10 percent reported external safety evaluations. This finding is a red flag for both users of AI agents and actors concerned about their societal impact. It underscores the need for more systematic testing of AI agents and more robust mechanisms for ensuring transparency and accountability. As a first step, governance institutions could establish and maintain an AI Agent Index of their own. But this alone would not be sufficient. The safety challenges presented by AI agents are many—and require a portfolio of governance responses.

Technical and Legal Infrastructure

Computer scientists have made headway in exploring technical mechanisms for mitigating risks from AI agents. These include requiring human approval for certain actions, automatically monitoring the behavior of agents, and enabling agents to be shut down in the event of malfunction or misconduct. Other proposals focus on creating IDs for AI agents, which would be accessible to users, auditors, and other key stakeholders, improving visibility into their operation and impact.

Drawing on the features of internet architecture, researchers suggest that governance infrastructure for AI agents should perform three core functions: attributing specific actions to particular AI agents, shaping the interactions among different AI agents, and detecting and addressing harmful actions. In practice, these functions could involve verifying that AI agents act on behalf of a particular individual or organization, establishing protocols for communication and cooperation between AI agents, and enabling certain actions of AI agents to be reversed.

Implementing these technical mechanisms raises a host of legal questions. For example, under contract law, will the actions of AI agents legally bind the users who instructed them? Under tort law, how will liability for harm caused by an AI agent be allocated among users, developers, and intermediaries? Can any of these actors be held criminally liable? If so, under what conditions and in which jurisdictions? And of course, how do AI regulations such as the EU AI Act affect the application of existing law to AI agents?

While significant research is needed to answer these questions, we are by no means starting with a blank slate. AI agents are not being developed in a legal vacuum, but in a complex tapestry of existing legal rules and principles. Studying these is necessary both to understand how legal institutions will respond to the advent of AI agents and, more importantly, to develop technical governance mechanisms that can operate hand in hand with existing legal frameworks.

Researchers are beginning to address the governance challenges from AI agents, but much work remains. Progress in technical governance requires legal knowledge, while effective legal frameworks require technical expertise. The governance of AI agents is a deeply interdisciplinary project. Computer scientists and legal scholars have the opportunity and responsibility to, together, shape the trajectory of this transformative technology. The stakes are high and the clock is ticking.