For the fifth year in a row, MIT Sloan Management Review and Boston Consulting Group (BCG) have assembled an international panel of AI experts that includes academics and practitioners to help us understand how responsible artificial intelligence is being implemented across organizations worldwide. In previous posts this year, we have explored artificial intelligence’s impact on the workforce, including why responsible AI requires more than training human experts to verify AI outputs.
This time, we asked our panel to react to the following provocation: Responsible governance that treats agents as autonomous decision makers will fail. On the surface, there is broad consensus, with a clear majority (72%) of our panelists agreeing or strongly agreeing with the statement. But digging deeper reveals a more nuanced conversation about what autonomy means, the relationship between autonomy and accountability, and what’s at risk when characterizing agents as “autonomous.” The picture that emerges: Calling agents autonomous decision makers risks allowing the humans and institutions behind them to evade responsibility for their actions. Effective governance, by contrast, ties every consequential decision back to a responsible party that can be held legally and morally accountable, in the context of a broader sociotechnical system.
Below, we share panelist insights and offer our practical recommendations for organizations thinking about agent autonomy through the lens of responsible AI governance.
Agents are increasingly “autonomous” — but only in an operational sense. Our experts acknowledge that agents are exhibiting a growing degree of technical or operational independence and ability to take action on their own. EnBW chief data officer Rainer Hoffmann says, “Agentic autonomy is real and growing,” while Renato Leite Monteiro, vice president of privacy, data protection, AI, and intellectual property at e&, observes that “self-improving agents are moving faster than we can map their failure modes.” Ben Dias, chief AI scientist at IAG, agrees that “agentic AI is rapidly moving beyond providing support or answers to taking autonomous action on our behalf.” For example, National University of Singapore vice provost Simon Chesterman points out that “agentic AI can plan, call tools, transact, and operate across workflows.” For these reasons, AI speaker and consultant Linda Leopold believes that “agents are autonomous decision makers, technically,” because “they act without constant human oversight and approval.”
But this kind of autonomy deserves closer scrutiny. Bruno Bioni, founder and director of Data Privacy Brasil, contends that “what looks like autonomy is [actually] delegated execution: selecting steps, using tools, acting within limits set by someone else.” Dias explains, “AI agents are given a goal and a set of guardrails, and then they independently determine and execute the sequence of actions required to achieve their given goal.” Amit Shah, CEO of Instalily.ai, similarly describes agents as “infrastructure that decides in the operational sense: It routes the order, moves the inventory, prices the risk, and so on.” As a result, Öykü Işik believes that “how we define autonomy in this context is critical.”
Operational autonomy does not translate into moral or legal accountability. For many of our experts, technical autonomy does not confer moral agency or responsibility. As Chesterman contends, “Autonomy in the engineering sense is not autonomy in the moral or legal sense.” Leopold cautions, “It gets problematic if we also start thinking of agents as autonomous in a moral sense — as entities with agency, or even coworkers, rather than the software systems they are.” Shah explains, “A machine can make the call, but it cannot own the outcome or consequence in the moral sense.” Jai Ganesh, Wipro’s former vice president of technology, agrees that “agents can make choices or execute actions, but they cannot be held accountable for the consequences.” For Carolina Aguerre, professor at Universidad Católica del Uruguay, “Responsibility is an inherently human faculty.”
Similarly, operational autonomy does not translate to legal responsibility. As Stanford CodeEx fellow Riyanka Roy Choudhury puts it, treating agents as autonomous decision makers “severs liability from capacity” since an agent “holds no assets to attach, no license to suspend, no deterrable interests.” Chow also points out that “agents have no legal standing and no assets,” adding, “They cannot be sued, pay damages, or be fully sanctioned.” Going further, Işik observes that “AI agents are stochastic and context-dependent,” not “coherent agents with stable intent” that meaningful accountability requires. For a recent example, Choudhury and others point to Moffatt v. Air Canada, in which a British Columbia tribunal rejected Air Canada’s argument that its chatbot was a separate legal entity accountable for its own misstatements. This case illustrates the challenge that companies face in governing agents that can act like human employees but cannot themselves be held morally or legally accountable.
Treating agents as autonomous decision makers undermines accountability. In fact, treating agents as autonomous creates an accountability vacuum and, as Bioni puts it, “imports a legal and moral status the technology has not earned.” As Chesterman cautions, “The more we speak as if agents ‘decide,’ the easier it becomes for firms and governments to launder responsibility through the machine: The model recommended, the agent acted, the human shrugged.” Or, as Bioni says, “it lets developers, deployers, and users hide behind ‘the AI decided’ whenever outcomes go wrong.” Even more bluntly, Shah calls the term autonomous decision maker “a governance fiction” that enables “blame laundering with better vocabulary.” For companies that remain accountable for the actions agents take, this accountability vacuum creates real risk if employees believe they can transfer blame and avoid responsibility.
The severity of this accountability vacuum depends on what’s on the line. For Monteiro, “Autonomy and accountability should not mix when the stakes are real” because regulators, boards of directors, and courts will require “a human they can hold responsible.” But while RAIght.ai co-CEO Richard Benjamins thinks that “impactful decisions should not be fully autonomously taken by AI agents,” he believes “trivial decisions can be.” Apollo Global Management’s AI lead Katia Walsh agrees that “for high-stakes decisions, responsible AI governance should not treat agents as autonomous decision makers, but for other contexts, it may be just fine” to treat them as if they were. And Aguerre contends, “Since not all AI agents perform activities with the same level of risk, the different levels of autonomy granted to an AI agent should be assessed against the tasks and objectives assigned.” Consultant Pierre-Yves Calloc’h sums it up: “Responsible AI means knowing exactly where autonomy must stop.”
The limits of autonomy depend on the sociotechnical context. Knowing where agent autonomy should end depends on the broader context of how humans interact with technology. Pointing to the example of autonomous driving, Australian National University’s Belona Sonna observes, “In many domains, AI agents are intentionally designed to make autonomous decisions because real-time operation demands it.” For Sonna, “The challenge is therefore not autonomy itself but ensuring that autonomous behavior remains aligned with ethical principles, human values, and its intended purpose.” Calloc’h says this distinction is particularly critical in high-stakes decisions, “where outcomes depend on trade-offs between conflicting objectives and implicit value judgments.” For him, these are “governance choices shaped by context, ethics, and strategy [that] cannot be reliably encoded or delegated.”
This is why several experts argue that governance should look past the agent to the system around it. Chesterman says, “The right unit of governance is not the agent as a little corporate citizen but the sociotechnical system in which it is embedded: the developer who built it, the enterprise that deployed it, the data and tools it can access, the permissions it has been given, and the humans or institutions that benefit from and remain accountable for its use.” Mark Surman, president of Mozilla, similarly explains that “agents don’t come from nowhere: People build them, companies deploy them, and someone profits from the decisions they make.” As a result, he urges organizations to “frame agents as extensions of human and institutional choices” since “the point isn’t to govern the robots [but] to keep humans accountable.” Finally, GovLab chief research and development officer Stefaan Verhulst argues that “governance must recognize [agents] as participants in broader sociotechnical systems shaped by institutions, data, incentives, legal frameworks, and community expectations.”
Recommendations
Considering the above, we offer the following recommendations for organizations seeking to responsibly integrate agents with varying degrees of operational autonomy:
1. Calibrate autonomy according to the stakes, not capabilities. Just because an agent can act autonomously doesn’t mean it should always be allowed to, especially where outcomes are hard to reverse or involve real trade-offs between competing values. Organizations should base delegation decisions on the reversibility and real-world impact of each action as well as the agents’ ability to accurately and reliably take action. Reassess those thresholds as the stakes of a task or goal change over time.
2. Enforce limits to autonomy by design, not by policy alone. After the lines between what an agent can execute independently and what requires human sign-off are drawn, build those limits into the system architecture itself (through properly scoped permissions, approval gates, hard stops, and technical controls), rather than relying on the agent or human operators to honor a written policy or prompt-based instruction in practice.
3. Name a human accountable for every decision. Regulators, courts, and boards need someone to hold responsible, and pointing to “the AI agent” is unlikely to cut it. Organizations should assign clear ownership for agent outcomes to specific roles or individuals, not to the technology, and they should do so before deployment rather than after something has gone wrong. In cases where agents operate across traditional business siloes, each department must understand their specific accountabilities and their responsibilities for oversight, escalation, and monitoring.
4. Govern the system, not the agent. Avoid accountability structures aimed at the model or agent. Governance should be directed at the full ecosystem in which the agent is developed, operates, and decides, including the developers who built it, the enterprise that deploys it, the humans who scoped and authorized its use, and the context in which it will operate. This ensures responsibility has somewhere real to land but remains shared across the entire workforce.
5. Create a culture of agent accountability. Interdependence keeps rising as autonomous agents increasingly coordinate with humans and other agents — and accountability becomes more muddled. Consider a group of humans, following guidance on how to work with autonomous agents, who produce a combined work product that leads to costly mistakes. Are the humans, human-machine teams, or governance system itself accountable? Specifying accountability as part of the design process and ensuring responsibilities are documented and understood avoids this lack of clarity. But holding employees accountable also means ensuring they can challenge agents and will be rewarded for raising concerns. This is critical to building an accountable culture, but an organization may be less able to demand those same responsibilities of end users when agents are supplied externally rather than deployed internally. For those cases, accountability by design is even more critical.