Over the last several years, artificial intelligence models have proliferated across the global economy. From scientific-research catalysts to email assistants to global-supply-chain management tools, AI products are seemingly everywhere.
But as businesses and organizations rush to implement these tools, leaders need to be aware of an important and persistent limitation of AI models: bias.
Kellogg Insight sat down with Kellogg professors Tessa Charlesworth and William Brady to discuss where bias lurks in AI, how consumers can bring a healthy degree of skepticism to interactions with AI models, and how regulation could help keep AI biases in check.
This conversation has been edited for length and clarity.
Kellogg INSIGHT: Where does bias in AI models come from?
Tessa CHARLESWORTH: Often when people approach bias in AI, they will try to tackle it from only one angle. They may point to annotators (the people who are label or tag data so that AI can learn from the data), or they’ll go after the biases that are already baked into the data itself. But, in reality, bias is coming in from many angles, all at once. We are interested in identifying all the places across the whole AI pipeline where biases—especially psychological biases—can creep in.
These biases can indeed be present in the training data and be manifested in how that training data talks about different groups like women versus men or gay people versus straight people. Biases can also be present in the choices that annotators make when tagging the data or tuning the algorithm. If your only annotators are white men in Silicon Valley, then they might not notice biases that others would see. And bias can also come in at the final outputs and consumption stages.
We have a new paper, for instance, that has one study where we ask people to choose what kind of AI models they want to interact with. Surprisingly, people are more likely to want to consume AI that reflects the world as it is (biases and all!) rather than consume or use AI that might challenge or uproot their biases. This means that people aren’t even auditing the biases in how they themselves are consuming AI.
William BRADY: And in each of those stages, there are other forms of bias that are getting trained in algorithms. Psychological biases that are baked into our cognition also get baked into AI systems even if the systems weren’t designed to measure them. In my work, I talk about our natural attraction to emotional, moral, or in-group information, which are types of information our brains evolved to pay attention to because it helps us solve social and environmental problems. If you design an AI system to optimize for engagement, for example, in practice you end up with an AI system that promotes emotional, moral, and in-group information because it learns that we tend to engage with that information. As we’ve seen in the past decade on social media, this can lead to negative social consequences—polarization, for example—rather than simply increasing clicks on a platform.
There are also forms of bias when it comes to how we as humans think of AI. What assumptions do we make about it when we seek information from it? A lot of people assume that AI is less biased than humans, which can lead to a level of trust in AI output. But if society’s skepticism toward AI grows when companies design AI systems in ways that don’t always serve public interest, the opposite can happen. So it’s also important to think about bias in terms of people’s tendencies when it comes to trusting AI systems.
INSIGHT: So there’s even a polarization of skepticism around AI, it sounds like?
CHARLESWORTH: Totally.
BRADY: There are group differences in how people view AI. Research shows men are more likely to adopt AI for certain tasks—and there are consequences to that. There are also differences in how you get evaluated if you report, “I use AI for this task.” There’s an entire sociological level of analysis where you could talk about bias, but people tend to focus on training data. But in fact, when we think about bias in AI, we should be thinking of what that means through all these levels of society now.
CHARLESWORTH: Yes, exactly. As someone who mostly studies the biases in the training data, I see this all the time. I myself used to think that once the training data goes into the actual algorithm, then it’s just going to directly transmit whatever biases were already in the training data. As we would say, “garbage in, garbage out.” When you approach a computer, we still have this outdated assumption that we’re getting objective outputs.
But with AI, it can be so much more than just direct transmission of training-data biases. There’s a potential for bias in data to become amplified through the algorithms themselves because of how they are optimized, as Billy’s research shows.
BRADY: As we’ve mentioned, bias can creep in at any point of the broad process of AI systems that involves accepting input and exporting output. This includes the general learning rules of AI systems, such as optimization. Anytime you choose, “I want to optimize for X,” that’s a biased decision: by definition, the system will amplify X at the expense of other things. And the reason why that’s important is because the people deciding what to optimize an AI system for have specific incentives that may not always be aligned with the incentives of the consumers of the AI system. Social-media algorithms are a great example of this.
CHARLESWORTH: This shows just how much human decisions get in the way of AI and alter what we might have otherwise seen as objective. This is just one example where humans are turning the knobs at every single stage, and each time they turn a knob, there is an opportunity for messy human biases to come into the system.
INSIGHT: What are some examples of bias that people should be concerned about?
CHARLESWORTH: Of course, there are all the biases we would think about first: The training data of AI often represents women as more passive than men, for example, not to mention all the biases attached to race, sexuality, ability, or health stigmas. We have all seen many examples of the failures of AI in equally representing, talking about, or creating pictures of different groups.
But there are other less-talked-about and, therefore, to me, more-concerning biases. One example would be the dominant language of AI. English training data make up the overwhelming majority of these models’ input. And English carries within it a whole bunch of assumptions and biases about social groups, like gender and race; sometimes even more strongly than other languages. English dominance alone is setting us way back in the bias space.
What’s more is that if you’re missing or underrepresenting other languages or the voices from other demographic groups, you’re going to be reproducing the biases of these dominant languages and groups across history. And when this reproduction of historical bias gets fed to us through AI, my fear is that it’s not just going to be like, “Ah, okay, yeah, this is just another outdated reprint of the dominant-group narrative.” Because we think of AI as objective, it’s easy to conclude that dominant-group narrative is right and true, in some way. That’s a major concern.
BRADY: Another thing we as a society should start worrying about—which stems from bias at various stages of developing AI systems—is “model collapse.” As the data that is being output into the internet starts feeding back in to retrain models, this can create a feedback loop where the model output starts to become more and more disconnected from reality. By definition, then, AI models might become more biased over time as they collapse into a representation of their own biased output.
INSIGHT: Are there ways that designers are being proactive in trying to reduce bias early in the process, or wherever they can intervene?
BRADY: There are certainly solutions. For instance, earlier I talked about the issue of companies deciding what to optimize in algorithms that do not always align with consumer interest. Of course, you can optimize algorithms for different things, theoretically, but you have to test that these things could reduce the types of bias we care about. Engagement-based algorithms, or algorithms that optimize engagement, lead to all kinds of biases in the information we consume because, in practice, they amplify the highly emotional and moralized information only produced by the most extreme people on the internet, or professionals who make a living off of things like “rage baiting.” In my research, we are asking whether we can improve people’s experiences on social media by reducing the undue influence of these extreme users to see how the information ecosystem becomes better for constructive civil discourse.
Most people assume there is an inherent trade-off: if I don’t optimize purely for engagement, people aren’t going to actually use social media because content will then be boring. In fact, we show that, while total engagement is slightly higher using the engagement-based algorithm, people report enjoying the platform more under our algorithm that reduces the influence of extreme users.
This gets at the tension between short-term user engagement and long-term user retention on the platform. That’s something companies could think about more. But most of them are hyper-focused on short-term clicks for advertising revenue. I think people over-rotate on the idea that short-term engagement is the only thing that keeps users online.
There is also the idea of giving users a choice in algorithmic and AI systems. Some platforms, like Bluesky, give users the option to choose which algorithm or feed they’re going to view. But in practice, most people just choose to use the default algorithm anyway. So even giving users the option doesn’t mean they’re adopting it.
And as social psychologist, I also highly doubt that giving users full choice is a way to reduce bias, if that’s your goal. You’re probably going to get potentially even worse echo chambers if you just give people full control of which information they see. But choice and user control can be a virtue in and of itself, so that is a larger conversation.
CHARLESWORTH: We can also tell you a bit about what likely won’t work.
INSIGHT: What does a reduced-bias AI look like? And is that achievable, or should we be thinking about other things, like representation?
CHARLESWORTH: These models are already far enough along that we’re not going to be able to change the input data. That horse has left the stable. And there’s also very little evidence that de-biasing the training data is the most efficient or effective way to reduce bias overall. Companies simply won’t have the incentives or interest in starting over from scratch and doing a labor-intensive intervention like changing the training data. Of course, we should do our best to increase representation, as I said, from non-English outlets, and from more minority voices.
But much bigger and more-effective changes will likely need to happen at the algorithmic, annotator, and consumer levels. As Billy said, this entails changing the algorithms about what gets optimized and amplified.
It will also be helpful to increase conversations or marketing to make consumers eager for less-biased AI. We can focus on humans’ interactions and on trying to educate them to be more-ethical consumers of AI. We have so much more literature on how to change humans’ attitudes and behaviors than we do on how to change AI itself, just because psychology has been around much longer than AI has been around. If we can start to make traction on how to make people aware of and concerned about the biases that they’re consuming, they may start to discount AI, rather than take it as objective truth. That, to me, seems like the more immediately tractable lever to pull.
Some of the big players in AI have started to do this. They have disclaimers, for instance, saying things like, “Be aware that I am trained on biased data, and I have these biased decisions.” It’s our job now to teach consumers to read those disclaimers and be aware of them.
BRADY: I definitely think a multipronged approach is good. We do need buffers, including AI literacy. I have some skepticism about our ability in general to be perfect critical consumers of AI. And we need data to demonstrate whether AI with less bias built into it is actually more enjoyable for consumers if we want to engage with companies that want to focus exclusively on revenue.
But I also think that, as consumers, we shouldn’t just wait for companies to make changes based on data. My concern is that when it comes to generative AI, which is getting widely adopted now, we’re basically standing on the precipice of where social-media companies were in, say, 2009, when they chose to prioritize scaling and increasing advertising revenue, without studying what the psychological effects were going to be of introducing engagement-based algorithms. Fast-forward to now, and some of the same people who developed those systems are in documentaries saying, “these decisions were reckless.”
When it comes to generative AI, we are at a critical juncture where we are at risk of repeating the same things that we saw with social media and its potential harms. We should demand that AI companies and politicians enforce safeguards that keep harms to society and the consumer in mind. I’m concerned with U.S. politicians who make public statements that AI safety is a thing of the past.
CHARLESWORTH: There are models like the EU, which is way further along than the U.S. in terms of regulation of AI.
BRADY: Regulation is often assumed to be anti-profit or innovation. But actually, the data definitely doesn’t speak to that.
INSIGHT: How can the designers of AI models improve user literacy for the information that they present?
CHARLESWORTH: For users, for researchers, and even for companies who are implementing “off-the-shelf” AI, the more transparent the AI, the better. Because then, when they’re implementing it into their workflows, they can actually understand what went into that cocktail of bias. If we could see more transparency, for example, in (a) what kind of data, exactly, was this model trained on, (b) what kind of training or checks or annotations did it go through, and by whom, and (c) how is it continuously updated or re-tuned over time—htat would tell us much more about the possible influences not only of the training-data bias, but also the algorithmic and repeated annotator biases.
Of course, this is important in everyday consumption, but it is also important for researchers and companies. For example, right now we’re seeing this huge move of psychology and social-sciences researchers adopting AI blindly, without knowing even how these models were constructed. And that’s a big problem. We would never do that when running a traditional experiment. We would never just apply a tool without knowing how it was constructed. Companies are doing it, too, by relying on off-the-shelf tools from other AI companies, even without knowing about the deeper training.
BRADY: If there’s no regulation, we rely on decisions by private companies, which are determined by private incentives.
INSIGHT: Is there a way of influencing those private decisions of companies to adopt AI?
BRADY: Regulation is critical. Regulation is often assumed to be anti-profit or -innovation. But actually, the data doesn’t say that. If there’s no regulation, we rely on private companies’ decisions, which are determined by private incentives.
CHARLESWORTH: But, at the same time, right now in the U.S., regulation, macro-level policy, legislation, and executive actions seem to me to be a nonstarter, unlike in Europe, where there is political will and an appetite. I mean, I wish we could rely on regulation, but I think we need to focus more on creating a consumer market for transparent models.
To give a research example, Hugging Face is a platform where a lot of AI models are uploaded. Every time you upload a model to the platform, you need to show its carbon footprint. That incentivizes community norms around creating models in the research space with a lower carbon footprint. If we could do something similar to reduce bias alongside the carbon footprint, we could start to push collective interests.
BRADY: We can also put even more pressure on democratically elected officials to work with AI companies toward regulation around AI safety. Yes, this can feel difficult in a highly polarized society like the U.S., but it can, and has, been done in the past.