For the fifth year in a row, MIT Sloan Management Review and Boston Consulting Group (BCG) have assembled an international panel of AI experts comprising academics and practitioners to help us understand how responsible artificial intelligence is being implemented across organizations worldwide. In previous posts this year, we have explored AI’s impact on the workforce, what human verification of AI outputs entails, and how to think about agent autonomy.
New research has emerged suggesting AI overuse or overreliance risks degrading our cognitive and moral judgment, resulting in “epistemic de-skilling” and reduced critical thinking skills. Evidence-based concerns about AI’s impacts on children’s cognitive and emotional development, in particular, led New York City to introduce the nation’s first ban on generative AI in schools.
Given the increasing salience of these worries, as a final provocation in this series, we asked our panel to react to the following: The defining challenge of responsible AI will be preserving trustworthy human judgment in the face of capable machines. The majority (65%) of our panelists agree or strongly agree with the statement, contending that human judgment is a muscle worth preserving, even if it is increasingly at risk of atrophy. However, a significant minority (19%) disagree or strongly disagree, and a further 15% remain neutral (neither agreeing nor disagreeing). The experts diverge on what human judgment means and how it differs from human oversight, and cite its inherent limitations. But beneath the surface, broader agreement appears to exist around this concept: The goal isn’t to preserve human judgment for its own sake but to design human-machine systems that keep trustworthy human judgment available when it matters most.
Below, we share panelist insights and offer practical recommendations for organizations looking to create systems that build in room for meaningful human judgment as AI grows more pervasive and capable.
Human judgment is a muscle worth preserving. Most of the experts believe that human judgment is worth preserving while machines’ capabilities increase. Ben Dias, IAG’s chief AI scientist, says, “The defining challenge of responsible AI will most certainly be preserving trustworthy human judgment as increasingly capable AI systems operate at greater speed and scale.” Similarly, Alyssa Lefaivre Škopac, Alberta Machine Intelligence Institute’s director of AI trust and safety, believes that “judgment, critical thinking, and the ability to evaluate AI outputs and actions will be the most important and differentiated skills going forward.”
For many of our panelists, human judgment is critical for providing context that machines lack. United Nations University rector Tshilidzi Marwala says, “AI can process information and generate recommendations at extraordinary speed, but capability does not equal wisdom, context, or legitimacy.” Consultant Pierre-Yves Calloc’h explains: “AI systems live in digital worlds, which are a partial representation of reality.” He adds, “Only a human living in the real world can assess whether an answer or recommended action generated from a prompt and the model’s training material is truly fit for purpose.” For Automation Anywhere’s Yan Chow, “human judgment matters most where machines are weak — novel situations, ethical trade-offs, context not captured in training of large language models.”
Many experts also contend that preserving human judgment is essential for maintaining human agency and accountability. “Human judgment, grounded in expertise and experience, is what lets us steer, verify, question, make trade-offs, and stay accountable when working with AI,” says consultant Linda Leopold, and serves as “the foundation of human agency.” But she also cautions that “judgment and critical thinking are the skills we risk losing if we rely too heavily on AI, which is why preserving them is essential.” Marwala agrees that the defining challenge of responsible AI will be ensuring that “humans retain meaningful judgment, agency, and accountability” as machines’ capability expands.
The answer is not simply to keep a human in the loop. Many experts also recognize the limits of human judgment. As GovLab chief research and development officer Stefan Verhulst notes, “Human judgment was never inherently trustworthy.” Chow explains that “humans face fatigue, bias, lack of experience or expertise, and poor calibration under uncertainty.” In fact, Instalily.ai CEO Amit Shah cautions that “keeping humans continuously in the loop can actually degrade quality by reintroducing inconsistency and error into decisions machines may be better equipped to make.” Going further, Yasodara Cordova, distinguished member of the investments committee of the Co-Develop fund, cautions that “human judgment has never been uniformly trustworthy, and building machines to defer to it risks encoding existing hierarchies of whose judgment counts as legitimate.”
Given the limits of human judgment, TÜV AI.Lab CEO Franziska Weindauer argues that “a human in the loop does not automatically make a system trustworthy.” For RAIght.ai co-CEO Richard Benjamins, “Human judgment is a bigger idea than oversight alone,” and what actually needs preserving is judgment that “serves humanity as a whole, not just the interests of particular groups.” Taking this a step further, Stanford CodeX fellow Riyanka Roy Choudhury argues that “neither human nor machine judgment should be presumed trustworthy.” For Choudhury, trustworthiness means that “no decision maker, human or artificial, should be the judge, jury, and final appeal.”
Growing AI reliance threatens to displace and erode trustworthy human judgment. Despite a consensus around the worth of preserving human judgment, our experts worry that a growing reliance on AI threatens its displacement and erosion due to “incentives to adopt, automate, and optimize,” as Škopac puts it. Dias observes that “as AI production outpaces the capacity for meaningful human oversight, human review becomes the bottleneck, [and] people are progressively removed from decisions.” And Renato Leite Monteiro, vice president of privacy, data protection, AI, and intellectual property at e&, worries that because “AI is anthropomorphized and agreeable by design,” we will “quietly stop valuing the judgment of the people we ask to supervise [it].” The risk, says Ranier Hoffmann, chief data officer at EnBW, “is that organizations design judgment out of the work while leaving humans nominally accountable.”
Experts also worry the trend of overreliance threatens moral and cognitive capacity. As Hoffmann observes, judgment “develops through consequential decisions, including wrong ones.” When machines decide, “they remove the ground on which judgment was trained,” he adds. Škopac agrees that “as more entry-level tasks get offloaded, so does the learning, muscle memory, and scar tissue that accompany these ‘tedious’ tasks that we understandably see an opportunity to automate.” Dias cautions that “prolonged human disengagement erodes the expertise needed to recognize failures and intervene effectively.” Researcher Kirtan Padh hopes we can use AI to improve “productivity, without losing our learning and critical thinking skills.”
Trustworthy systems focus on human-machine collaboration. In the end, Shah says, “the defining challenge of responsible AI is not preserving human judgment for its own sake.” Instead, as AI grows more capable, ADP’s chief product owner Naomi Lariviere believes, the challenge will be figuring out “where human judgment actually matters.” For Shah, “human judgment belongs primarily upstream: setting objectives and guardrails, defining thresholds for autonomous action, determining when uncertainty or out-of-distribution cases require escalation, and assigning responsibility for outcomes.” For Wharton School professor Kartik Hosanagar and others, it matters most for “mission-critical tasks” and in “high-stakes settings.”
For many experts, preserving trustworthy human judgment when it counts is a matter of design. Because “humans and machines excel in different scenarios,” Chow contends we must build “systems that recognize that difference.” Verhulst agrees we should design “human-machine judgment environments in which each compensates for the weaknesses of the other.” Škopac is optimistic that “after a few scraped knees, it’s going to become clear that completely offloading to AI results in worse outcomes, low morale, and more embarrassing headlines.” If we strike the right balance, Dias believes, we can “preserve human accountability, competence, and authority, while maximizing the value extracted from AI.”
Recommendations
Considering the above, we offer the following recommendations for organizations seeking to design systems for human-machine collaboration that preserve the opportunity for trustworthy human judgment:
1. Identify where human judgment matters. Not every decision requires human review or intervention. Organizations should identify where human judgment matters in their specific decision-making processes and workflows, and focus on preserving the opportunity for human judgment in the context of high-stakes and low-precedent decisions.
What’s more, human judgment will continue to be essential in determining the philosophical bases on which human-AI machines operate. What is the purpose of a given application? What actions do we allow the system to take autonomously? What is appropriate evidence for a decision — that is, when do we know enough to make a decision? These questions should be foundational to the design of every AI system, presenting the earliest embodiment of human judgment and accountability for AI. As Shah remarks, human judgment is critical to these “upstream” concerns.
2. Create the opportunity for human judgment by design. Where human judgment matters, the opportunity to exercise it must be designed into the system, not assumed that it will exist. Simply keeping a human in the loop is not enough here. A human asked to review or approve an outcome requires the context and evidence necessary to make an informed judgment. And human review cannot occur after the fact when there is little practical ability to change the outcome. Organizations should build the opportunity for real human review and intervention into the system, process, or workflow from the outset, so that when judgment is needed, there’s an actual point at which it can be exercised.
3. Use it or lose it. Judgment is a skill that will atrophy if it isn’t exercised, and the high-stakes, low-precedent decisions where human judgment matters most are too rare to provide the only training ground. Organizations should deliberately build in opportunities to exercise judgment outside of high-stakes moments, through simulations, case studies, and postmortem or retrospective exercises. They should go a step further, deliberately keeping some lower-stakes decisions in the hands of humans instead of offloading to AI as practice for when it counts. Retaining human decision-making will carry a small near-term opportunity cost but will have net benefits over the long term by building critical organizational capacity.
4. Calibrate training by level. How judgment is honed should look different depending on experience level. Junior team members need chances to build judgment they haven’t yet developed, while senior decision makers have established judgment but fewer opportunities to exercise it when they rely more on AI-assisted tools or delegate routine decisions. Organizations should design distinct pathways for individuals of different levels of experience.
5. Build feedback loops. Neither human nor machine judgment should be trusted by default. Organizations should track outcomes on both sides, such as when the human override of an AI recommendation was right or wrong, or when deference to the AI output was right or wrong. This record can help organizations recalibrate where responsibility should sit, since human-machine collaboration is something that will require ongoing adjustments as AI capabilities evolve. It will also be critical to assess whether human oversight is effective.
Do human reviewers blindly follow the AI recommendations or do they exhibit true judgment? This analysis will identify areas that may require redesign, additional training, or other interventions.