X Previews Labels That Tell Users Why Its AI System Flagged Their Posts

X’s AI-powered content moderation system once mistakenly labeled a post featuring an image of a SpaceX rocket launch as “adult content.” No one, including the author of the post, could see why. That’s the problem Michael O’Herlihy, a Twitter veteran who recently rejoined X as head of safety, wants to fix. In the future, X’s content moderation systems may show their decisioning process, he suggested at the Brand & AI Safety Summit in New York on Tuesday (Nov. 10). There, O’Herlihy previewed designs for labels displayed beneath content on X that explain what its AI detected in a post, detail how exactly the post’s reach was limited after being flagged, and allow users to appeal the decision. In his remarks, O’Herlihy took the SpaceX rocket launch to a different theoretical outcome, in which X’s content moderation system might have mislabeled the post as “graphic violence” by misinterpreting the rocket’s exhaust as an explosion. With more sophisticated labels that show why a post was flagged and that let users appeal, less content could be mislabeled on the platform, he suggested. “With that additional context, we can take another look and fix it if we got it wrong,” he said. He added that people might disagree with the explanation, “and that's fine." "Agreement was never the test," he said. "What we want to hear is, ‘I still disagree, but I see why you got there.’ Or better still, ‘Oh, you were right.’” X declined to specify whether the expanded content labels, which O’Herlihy called a “design preview,” are in testing or when they might be rolled out. The SpaceX mislabeling snafu was a relatively anodyne example of X’s content moderation getting it wrong. Just last month, researchers at The Center for the Study of Organized Hate reported 464 posts from U.S. accounts this year that included dehumanizing content and violent threats against Muslims. It found that X’s policies prohibit every content category identified in the posts, yet they stayed visible for months, despite automated checks. Sixty-two percent of the posts remained accessible even after manual reporting. An X spokesperson did not respond to a request for comment on the findings. AI-powered moderation X, formerly Twitter, has been using automation to aid in content moderation for over a decade. In a filing with California’s attorney general earlier this year, X said it combines machine learning models, heuristics, and large language models to identify and moderate content that breaches its rules. But X has previously pushed back against requirements to disclose information about how it policies content. It sued New York’s attorney general last year over a state law requiring platforms to report on their moderation policies, arguing it violated the First Amendment. In August, a federal judge ruled against the company, requiring it to publicize its content moderation policies.O’Herlihy, who rejoined X after that suit was filed, struck a different note. He argued that the shift to AI-led moderation is more effective than human review, because when humans review a case, “and at the end of it, they click a link and they throw away all of their work,” he said. “If I come back and ask somebody three weeks later, ‘Why did you make this decision?’ They will relitigate, and they may or may not come up with the same outcome," he continued. AI, on the other hand, provides a "paper trail that can be audited." “The AI won't always be right, but we'll be able to understand why it made the mistake, and we will learn from that and evolve,” he said.O’Herlihy added that he would rather machines handle the most disturbing work, like child safety cases. “I don’t want humans to have to do that type of work,” he said. Automation can also help accelerate cleanup, he suggested. A post on X gets half of its total impressions between two and three hours after publishing, but most safety teams work on a 24-hour turnaround. AI shortens the workflow. A glass box In his new role at X, O’Herlihy suggested his top goal is to make decisions about safety not just more explicit using AI, but more defensible. “Can this decision be justified under the rules? Transparency provides the evidence," he said. "Can someone see how we got here? Verification is the test.” O’Herlihy spelled out a vision for a “glass box” at X to replace platforms' typical black box approach to content moderation. It includes four walls: transparency into how content ranks on X, which can be done thanks to its open-sourcing of the For You timeline code; context, provided by crowdsourced fact-checking from Community Notes; specific account restriction details provided via X’s Under the Hood product; and, eventually, explanations of individual content decisions. Only that last piece isn’t live yet. But even in this vision, the “glass box” won't be entirely transparent, O’Herlihy acknowledged, saying it “shows the rules and the decisions, not the answer key.” X does not disclose the signals or thresholds it uses to catch scams, spam, or coordinated abuse in an effort to keep the system “open enough to verify, closed enough that it can’t be gamed.” Independent verification will help prove whether the system works “consistently at scale,” according to O’Herlihy. X lets advertisers use DoubleVerify and Integral Ad Science to help ensure their messages don’t show up next to undesirable content. The pitch arrives after years of strain between the platform and advertisers, who, prior to Musk’s 2022 acquisition, supplied nearly 90% of the site’s revenue. But following his takeover, the Tesla and SpaceX boss took a hammer to the platform, halving headcount, upending account verification rules, and gutting its content moderation operations. The changes catalyzed an advertiser exodus, with big brands like Coca-Cola, Unilever, and Ford pulling spend. At the 2023 DealBook conference, Musk told boycotting advertisers to “Go fuck yourself.” In 2024, X sued the World Federation of Advertisers and its brand safety initiative, the Global Alliance for Responsible Media (GARM), alleging that it and some of its members colluded to boycott the platform. The scuffle led to the hurried closure of GARM, and the two sides finally settled earlier this year. X’s ad business has roughly halved since Musk’s buyout. Parent company SpaceX reported $538 million in advertising revenue for the third quarter, compared with roughly $1.14 billion in the same quarter of 2021, Twitter’s last full year as a public company. Though the platform’s ad revenue grew 18% year-over-year in the most recent quarter, it’s down 6% for the year-to-date. For a platform still fighting to win back big brands, a safety pitch built on transparency is also a sales pitch. On stage, the new safety chief urged advertisers to embrace independent measurement and demand greater accountability from all media platforms. “Safety shouldn’t be something platforms ask you to trust,” he told the room. “It should be something you can verify.” His direct call to action for advertisers: “Make ‘Can I check?’ a normal question. Ask every platform, including us.”

Social Media

X Previews Labels That Tell Users Why Its AI System Flagged Their Posts

The company’s new head of safety wants the platform to “show the work” on content moderation decisions

Inside This Story

Key Takeaways

UNLOCK FULL ACCESS

Kendra Barnett

Media

OpenAI’s Monetization Chief to Ad Partners: User Trust Means Some Things ‘We Just Can’t’ Do

Pastor Talarico, 3 Billion Views, and $56 Million on Broadcast: How James Talarico's Team Flipped Texas

The 2026 Political Ad Outlay Was the Dumbest Money Spent In the History of Advertising

Democrats Learned to Win the Social Feed After 2024's Digital Drubbing

Canva’s Pricing Puzzle Is Every Marketer’s Problem

Who What Wear Sells AI Visibility to Brands, Helping Fuel Double-Digit Revenue Growth

ADWEEK Tech Advantage: Brands Are Showing Up in ChatGPT. Proving It Pays Is Harder.

Anthropic Strikes Its First Licensing Deal, With the AP, for Election Coverage

Martech Entrepreneur Adam Robinson Among Nine Dead in Florida Plane Crash