SUMMARYDeb Roy of MIT argues that AI-generated text in civic life must remain tied to accountable human authors, especially as realistic machine-written public comments make it harder to trust democratic feedback channels. In a Bulletin of the Atomic Scientists essay, he calls for “standing” and “answerability” in communication and describes design principles for tracing AI-assisted output back to people who can take responsibility. His Center for Constructive Communication is also building tools like VoiceTrace and running dialogue programs to support trustworthy group deliberation.
When government officials craft policies, there is often a comment period for public feedback. Lawmakers have long had to sort through fake feedback that is mass-produced in an effort to sway policy, but they’ve generally gotten used to detecting it. With, artificial intelligence, however, people can submit realistic-seeming comments, which are harder to dismiss as being fabrications with no person behind them. That’s just one way AI can be used to generate text and statements, in civic conversations, for which no one can be found to be answerable.
This issue, among others, has caught the attention of Deb Roy, professor of media arts and sciences at MIT and director of the MIT Center for Constructive Communication (CCC). In a new essay published in the Bulletin of the Atomic Scientists, titled, “Who stands behind the words?,” Roy calls for a renewed emphasis on responsibility in communication, asserting that even when AI is used to craft text, there needs to be accountability and “answerability.”
Building better group dialogues to produce more public trust is a core part of the work of CCC. The group, along with Cortico, the closely affiliated nonprofit Roy co-founded, has helped facilitate group discussions on important issues in many situations — including civic meetings, schools, deliberative assemblies, and private organizations — and used tech tools, including AI, to help find themes among comments, and more. But the center also designs this process so that participants can check and override the tech-based output at each key step. Ultimately, a person is responsible for every public comment. MIT News spoke to Roy about the essay and the work of the center.
Q: You’ve identified this problem about the lack of accountability behind words that are generated by AI. When did you really first notice this?
A: It came into focus when Chat-GPT [which is powered by a large language model, or LLM] launched. As for my own history with language models, my PhD work explored what we might now call PLMs — Puny Language Models — so I’m quite familiar with language models and have tracked the field’s rapid evolution. But Chat-GPT turned a corner. Two things changed at the same time: the depth of the model and its knowledge, and the way you could interact with it.
I became obsessed with trying to find the limits of what it could do. And I felt that no matter how hard I tried to stretch my arms around it, I couldn’t. I built my tenure case at MIT on language grounding — the idea that it’s not enough for an AI system to just process lots of words and other symbols. I believed that to understand what words mean, you actually need a physical body and life experiences as part of the process. So I revisited the philosophy-of-AI literature I had once been steeped in, which asks what fundamentally distinguishes the kind of system that large language models exemplify, in extraordinarily advanced form, from the kind of processing we humans engage in. I gave several presentations to my own graduate students trying to make the case that, even though AI seems so intelligent, it’s actually different in kind — and I could not convince them.
And I thought, I know the difference, but I’m having trouble convincing even myself of what it is. And that led me to ask, “Wait a minute: Is there someone or something that stands behind the words, a human responsible for what is being said?” That’s the thing this extraordinary new language machine doesn’t have.
Whether these systems genuinely understand language is actively debated. But even if you grant them all the comprehension you like, they still lack standing. The buck can’t stop at a thing, a software system, with nothing to lose.
Q: The Bulletin piece has these interesting examples of civic feedback forums where it looks like there are real people saying things, and instead you have a barrage of comments from AI. How did you find some of these other examples in society where things go off the rails in this way?
A: After the release of ChatGPT, I started hearing a concern from people inside government who review public comments on proposed rules: a suspicion that not all of the comments were written by humans.
My reaction was: That’s going to delegitimize this channel for public feedback. That was several years ago, but everyone now knows this is happening. Public comment is a pillar of democracy. You don’t just put a new rule in place. People need to have a voice and feel like they’ve been heard before a rule is imposed on them. If you skip that step, people don’t accept rules as legitimate. That’s why that process is baked into how America works. And that pillar just cracked, and by now, in my mind, it’s fallen over. Has anyone noticed? My sense is the people on the receiving end of these comments have quietly adjusted the weight they give to public comments. It was never perfect, but boy, now much of its value is being swept away.
I open my essay with this example: In 2017, the Federal Communications Commission received more than 22 million public comments on its proposal to repeal net neutrality rules, and the New York attorney general later found that nearly 18 million of them were fake. As I write in the essay, “That fraud was caught because it left fingerprints: templated text, duplicated phrasing.” But large language models erase the fingerprints. Now every comment can be fluent and different, and reviewers can no longer tell which ones express views that real people formed and stand behind.
If you have enough of the stuff that’s fake — words that no one with a stake actually stands behind — it desensitizes you to what’s real. That filters back to the public, and people realize: If I take the time to express myself, they’re not even listening any more.
Q: Could you say a little bit more about your point that we need people with “standing,” who will take “accountability” for all pieces of text, even when those bits of text are generated by AI?
A: In the legal system, standing is a formal idea: sometimes a case isn’t heard because the people bringing it don’t have standing. But informally we use that term all the time. I’m not trying to formalize standing in everyday life. Instead I’m asking: What is the informal sense of the term, borrowed from the legal one, that we need to pay more attention to now? Here in New England, consider the open town meeting: The person who stands up to speak is generally expected to have standing, as someone from that town with a stake in what gets decided. And if it’s a stranger who drove from across the state, or from out of state, everyone’s like: Wait a minute, who is that? They’re not from around here. In general, there’s just enough friction and it’s costly enough to show up somewhere you don’t actually have standing, that it doesn’t happen much.
Now LLMs and the current digital networked environment collapse those boundaries. We always took for granted that if someone has a voice, it’s because they have standing. They have some basis to be heard. We need to pay attention to this, and really consider how to design systems so that the people with standing are the ones who get heard.
The other idea I tried to develop is “answerability.” It isn’t formalized in law, but it’s baked into the legal process. If you have standing to give testimony, there you are, ready for cross-examination. You have to be able to answer for what you’re saying. And if you’re making a specific accusation about someone, you have to be willing to back it with facts. Standing and answerability have to travel together: The right to be heard comes with the obligation to answer for what you say.
That doesn’t mean everyone has to attach their name to everything. I write in the Bulletin essay, “Privacy and answerability aren’t opposites. … An institution can verify a person and stand behind their words without exposing them.” To voice a view, you only need to show that a distinct, eligible person is speaking. An accusation asks more: Someone trusted, like a reporter, has to be able to know who is speaking. As I say: “Disclosure obligations should scale with power, and identity protections with vulnerability.”
None of this means AI can’t help. AI can do a lot of the work inside the chain of accountability — drafting, summarizing, finding patterns. If someone with a stake in a rule uses AI to draft a comment, adopts it as their own, and is ready to answer for it, that’s legitimate, in this context. What AI can’t do is anchor the chain. There has to be a person at the end who stands behind the words.
In the essay I lay out four design principles for building that in: “attributable authorship,” a “decomposed and inspectable process,” the use of “provenance as infrastructure,” and the need to “widen standing rather than concentrating it.” As I write, the “animating rule is that autonomy is permitted while unaccountability is not.” And the metric that matters most isn’t the efficiency gained by removing humans, but the number of people AI places into roles where they can answer for what they do.
Q: There is a phrase in the documents of the Center for Constructive Communication, that we need things to move “at the speed of trust,” not at the “speed of tech.” And that implies really thinking about building up these systems. Could you just say a little more about what that phrase means to you?
A: I learned that phrase when I was in Appleton, Wisconsin, with Katherine Cramer [of the University of Wisconsin], a longtime collaborator, talking with community organizers who were running “Dialogue across Divides” conversations there. They had hundreds of people involved. I thought I was developing technology tools with exactly that same purpose, and was so excited to see it. And I asked one of the people who ran that organization, “How fast did you scale this?” And he paused, and then he said, “At the speed of trust.”
And I took that to mean “slow.” Then I realized he was saying: For you, slow, because you haven’t yet earned our trust. And I’ve since learned that when you do have trust, things can move fast. So bringing in technology is really a process of engagement. It doesn’t work to say, “Trust us, we’re doing this because we think it’s for your own good.” People will turn to someone they already trust and ask, “Is this good for us?” And if that person says “Yes,” things can move fast.
Look at what’s happening at the intersection of AI and society: The speed of AI and the speed of democracy are running on different clocks. Talk to people who work in democratic practice and they say, “AI must slow down, because democracy can’t move that fast.” Talk to people in AI and they say, “AI cannot slow down, democracy needs to speed up.” If we can’t get these two time scales in sync, we’re headed for a breakway situation, where AI gets deployed faster than democratic processes can respond, and decisions get made without the public. That’s starting to happen right now.
Q: We need people actively doing things to build dialogue and figure out how tech tools can interact with that. At the Center for Constructive Communication, you’ve done a few iterations of the center’s Real Talk initiatives to build civic dialogues, as well as the Youth Voices Initiative in school settings, and currently you’re working on the New Social Forms project. Could you say a few words about them?
A: The Real Talk effort represents the longest-standing thread in this work. A huge emphasis of our work has been developing better ways to listen to one another. We create a context that is always small — small is beautiful here — where people are willing to share authentically. And we have the consent architecture and the technology to reveal the larger patterns, and then share them. We’ve done this with hundreds of communities around the country. And closer to home, we’ve brought the same approach to our campus here at MIT through the realtalk@MIT program, with support from the Office of the President and the Office of the Provost, to help our community build a culture around dialogue and listening.
The second thing I’d emphasize is that we’ve been doing more and more work on deliberative processes and citizen assemblies, and on how to make them more accessible. What if a group of high schoolers says, “We have a real problem with our cellphone policy?” Suppose you want to have an assembly that brings students, teachers, and administrators together, not just to listen to people’s concerns but to make policy recommendations. So we did that. The center worked with a high school in New York City, where technology helped people reveal their preferences and differences and listen to each other.
We’re also building answerability into the technology. A prototype called VoiceTrace, developed by PhD student Cassandra Overney in our group, links every output statement — a summary of what was heard in a listening process, or a recommendation coming out of a deliberation — to the specific voices behind it, so any claim can be traced back to the conversations it came from.
The New Social Forms project pertains to a very interesting pattern emerging in workplaces around the world, certainly the ones we’re connected to. AI is an extraordinary tool for individual empowerment, but in the words of one of our collaborators at the Google Experience Institute, people are tunneling. They go deep, using AI to develop refined ideas, but when they surface, their ideas have diverged from their teammates’. So groups and teams trying to work together with AI, there’s a human coordination problem.
We’re going to expand the initial work we’ve started with Google because there’s a lot of interest in collective communication, trust, and interaction among people trying to work together.
So is that a civic context? It’s a workplace, but workplaces are civic spaces, too. They govern people’s daily lives, they’re where many of us build civic skills and learn to cooperate across difference, and there’s a long tradition of civic life rooted in shared work. We’re all humans trying to interact with each other. These are the same issues. Justice Louis Brandeis once used the phrase “industrial democracy” to capture this. When workplaces lacked certain kinds of civic processes and civility — two interestingly related terms — labor disputes used to turn violent. Procedural justice, mediators, and people hearing one another are just as necessary in the workplace. In the end, it’s all about human beings. Hearing the humanity in one another is what democracy depends on, and it’s just as necessary for communities and organizations to function.