Publishers Are Preparing to Opt Out of Google Search

This story was originally published in On Background with Mark Stenberg, a free, weekly newsletter that explores the key themes shaping the media industry. You can sign up for it here. For decades, publishers have done everything in their power, from the legal to the not-explicitly illegal, to rank as highly in Google Search as possible. For many websites, traffic from the search engine was their single greatest source of audience and, as a result, revenue. Now though, a handful of influential players in the digital media ecosystem have begun moving in the opposite direction, laying the groundwork for what was once unthinkable: removing themselves from Google Search. Last week, the content delivery network Cloudflare, which hosts roughly one-fifth of the websites in the world, gave Google an ultimatum. Beginning Sept. 15, all new websites signing up for Cloudflare, as well as all the customers on its free tier, will have the default settings in their bot management protocol set to block “multi-purpose crawlers” on any webpage that has ads. This means that any crawler that scrapes for both search indexing and AI training will be turned away at the door, unless the site owner decides otherwise. “We’ve been clear about what we want,” said Cloudflare chief strategy officer Stephanie Cohen. “We want a technical solution that allows you to be discoverable without having to give your content away for free.” While a handful of crawlers fit this description—Apple and Bing, among others—the primary, unnamed target of this action is Google, which infamously uses one crawler to both index sites and train its AI models. In doing so, Google forces publishers to make an impossible choice: They either allow both functions, enabling Google to scrape their content to train the AI products that are regurgitating their data without compensation; or they shut off both functions and disappear from Google Search, presumably losing their largest source of traffic in the process. To be fair, Google recently introduced an option called Google Extended, which nominally allows publishers to opt out of AI training without disappearing from Search. But publishers are wary that the program will penalize their search visibility, according to executives at two media companies. Similarly, new controls introduced in the U.K. enable publishers greater agency over what Google can do with their scraped content, but publishers are similarly hesitant to trust a solution that relies on the discretion of Google rather than an outright block of its crawler. "We provide web publishers with clear, granular controls to manage their content, including Google-Extended for AI training and a new Search Console control we are testing for generative AI Search features, neither of which impact traditional Search visibility," said a Google spokesperson in a statement. "We are committed to designing AI experiences that highlight the web, drive valuable traffic to publishers, and provide the insights they need to succeed." Historically, abandoning Google Search would have been commercial suicide, according to SEO consultant Lily Ray. It is simply too valuable of a source of audience discovery and traffic. “It’s a really hard tradeoff. Some publishers have already blocked OpenAI until it strikes partnerships, but with Google it’s hard,” Ray said. “Google is a different conversation because it has so many more users than other AI firms.” But the gradual erosion of search traffic in recent years has, paradoxically, given publishers more agency to consider walking away from the platform. USA Today Inc., which encompasses not just USA Today but a nationwide network of news sites, is weighing its options on the matter, according to CEO Mike Reed. The company, like many others, has responded to declines in search traffic by bolstering audience from other sources, like newsletters, social media, and events. Its traffic has remained relatively stable in recent years, hitting its goal of 1 billion pageviews every month for the last three years, according to Reed. Still, its monetization strategy going forward as it relates to AI will come from licensing agreements, which the company has struck with Meta, Microsoft, and Amazon, among others. As a result, USA Today Inc. is prepared to delist from Google in the next six to twelve months, according to Reed. Likewise, the creator network Beehiiv announced in a recent partnership with Cloudflare that its network of creators now has the ability to block the Google crawler. “I wouldn’t call it a big decision because we’re blocking other crawlers,” Reed said. “For those with licensing agreements, they get our content. For those without, we block them.” While USA Today Inc., Beehiiv, and Cloudflare are the first major players to take this step, executives at every major media company have a model for what it would look like if they blocked the Google Bot, according to one executive who wished to remain anonymous because of business engagements with Google. The decision of when to make that call is mostly a matter of math: Once search traffic drops below a certain threshold, the value of appearing there becomes less than the value of withholding that content as a bargaining tactic. And to be clear, every publisher would prefer Google to come to the negotiating table. But the alternative is bleak. If Google refuses to strike licensing agreements, more premium publishers could follow suit in denying its crawlers access. Doing so would degrade the quality of search results, making it harder for consumers to find accurate news. Just as social media has become a cesspool of misinformation, the open web could be overcome with untrustworthy content too. “Our content is showing up and powering generative experiences on Google at least hundreds of millions of times a day,” said the media executive. “If we weren’t in it, every one of those experiences would be worse.” Still, executives at other media companies are not so sure. The existing deal is a lose-lose situation for publishers: Allowing Google to scrape their content for free is self-defeating, but opting out has no clear material benefit either, according to one media operator who also wished to remain anonymous because of business dealings with the company. Plus, if a company opts out now, there is no telling what negative externalities that could create in the future, the operator added. Google is not compensating publishers for their data now, but if it does in the future, could those payouts be harmed by restricting the Google crawler in the interim? The options are unappealing. Nonetheless, that publishers are considering opting out of Google Search marks a symbolic milestone in the lifecycle of the open web. The posturing is all part of a broader negotiation, to be sure, but the threats are no longer empty. No publisher wants to make do without Google, but many are increasingly prepared to do so. Google, on the other hand, has no such backup plan. Correction: This story has been updated to include information on Google Extended and the new data controls available to websites in the U.K. A sentence stating that Google did not have any licensing deals with publishers has been removed, because the company has struck some deals, spanning a variety of forms, with several publishers. Talking Heds Newsweek Weakens (SCOOP): Traffic has fallen across the industry, but few have been hit harder than Newsweek, whose readership has declined from 100 million in May 2025 to 23 million last month, a drop of nearly 75%, according to Similarweb data. These declines have taken their toll: The company saw sales and product layoffs in June, rankings team layoffs in March, and video team layoffs in May, according to people familiar with the matter. Two key executives, the head of events Megan Knapp and chief product officer Bharat Krish, have both left the company in recent months. Last January, I chronicled the unlikely turnaround story Newsweek CEO Dev Pragad had authored at the outlet, based largely on the strength of its programmatic advertising business. With audiences dwindling, the business has contracted in tandem. The YouTubification of Netflix: On Tuesday, Netflix announced a sweeping new series of deals with digital publishers from Condé Nast, People Inc., Hearst Magazines, and more. The licensing arrangements will see premium video produced by the publishers brought to Netflix, meaning fans can find series like Walking Tours from Architectural Digest and Struggle Meals from Tastemade on the streaming service. The move is yet another foray from Netflix into bringing digital video into its ecosystem, following its tie-up last year to bring Ringer and Barstool podcasts onto the platform. So far, the company has stopped short of bringing user-generated content inside its gates, but this deal brings it one step closer. Video content from these publishers matches the Netflix pedigree and ensures a brand safe environment for advertisers, but it furthers the transformation of Netflix into a subscription-supported YouTube. Business Outsider (SCOOP): When Axel Springer executive Christian Baesler took over as interim CEO of Business Insider in May, he made it clear that his tenure as top lieutenant would be temporary. But given his wealth of experience atop media organizations—Baesler served as the COO of BuzzFeed and CEO of Complex, among other roles—it was not exactly clear why. It turns out that the German executive is raising funds for a separate project, according to two people familiar with the matter. I am told that Baesler has put those efforts on the shelf while leading BI, but that when Axel Springer names a new CEO, which is set to happen in the coming months, Baesler will resume his fundraising and recommit his focus to the new venture. I have scant few details about the project—Baesler did not respond to my request for comment—but given his track record, media would be my first bet. Weather Vanes: Yahoo is planning to relaunch its Weather App in the coming weeks, accompanied by a buzzy new marketing campaign to promote the update. The effort is the latest in a series of new, weather-centric offerings to debut from publishers, which are presumably interested in capitalizing on users’ daily habit of checking the forecast. CNN has made weather a central tenet of its new digital offering, and Byron Allen, who owns the Weather Channel, aims to plug BuzzFeed into its Local Now service. While I enjoy checking the weather as much as the next guy, I have yet to wrap my head around how such a service could offer anything more than incremental value to a preexisting product. The real weather heads have their own universe of hyper-scientific services, while most casual users can make do with the built-in Weather app in their phones. If anyone can explain this uptick in investment, please do. After School at Cannes (PODCAST): I spoke with Casey Lewis, author of the Substack newsletter After School, at Cannes the other week, and the kindly folks at Sounds Profitable recorded the interview, which you can watch in full here. Casey, who covers youth trends, talked about the decline of text-based media among younger audiences, the performative rise in analog hobbies, and how she anticipates creators growing their businesses in the coming years. Side Projects at Cannes (PODCAST): Similarly, I joined Clare Malley and Eli Williams, hosts of the Day One Agency podcast Side Projects, at Cannes to talk about what I was seeing on the ground. I made the case that Cannes is not the place for launching products, but rather where industry leaders come to build consensus about what is and is not working in the space. We talked about the declining focus on creative, the rising focus on creators, and why executives are finally admitting behind closed doors that artificial intelligence is not worth the cost. You can listen here. Quote/Unquote At Cannes, I moderated a one-on-one panel with Mike Peralta, the chief revenue officer of Future, a publicly traded digital media company that houses over 130 editorial brands, including Marie Claire, Tom’s Guide, and Who What Wear, among others. In the conversation, Mike and I discussed how Future is responding to the challenges posed by AI, especially as it relates to declining traffic, as well as how the company is working to be compensated for the use of its data and improving its answer engine optimization. This interview has been edited. Mark Stenberg: Should part of publisher revenue come from AI companies themselves? Mike Peralta: AI is nothing but a hallucination without our content—not just Future's content, but every publisher's. It's incumbent upon all of us to ensure there's the correct value exchange. Every publisher is playing both offense and defense: Our content has been scraped for years, and now it's a bit of an arms race. But original content is super important, independent publishers are super important, and without us, the internet's not quite as interesting. Mark: What has Future learned about showing up in answer engines? Mike: What works within the LLMs is authority and trust. If an editor with a real following writes something, that increases the citation. In the end, what did SEO do? It drove clicks to your site. What does the LLM do? It gives you a recommendation. So as a publisher, how can I show up in that one place? We're still early in figuring out how a citation aligns to an actual outcome. Mark: What does AI-driven traffic loss mean for publishers? Mike: Audience is probably going down for a majority of publishers. That being said, the audience you do have is probably 10 times better. They're more passionate—if you're interested in golf, you're going to keep coming to Golf Monthly. Mark: What should we watch for over the next year? Mike: Agentic. When my college-age son can create his own agents better than I can, I better start paying attention. Agent-to-agent buying and selling is going to come up faster than we think. A year from now, more than half of programmatic buying is probably going to be done through agents. Pulled Quotes "Gambling has become a more common leisure activity than reading a book: Last year, 57 percent of Americans placed a bet."The Atlantic’s Rose Horowitch, on the end of readingREAD MORE "Even if you raise $50 million, no one’s gonna write about you unless you’re doing something crazy."Startup founder Zuhair Lakhani, on ragebaiting as marketingREAD MORE “I think the demand for philosophers with A.I. training is, if anything, outstripping the supply right now.”NYU professor David Chalmers, on philosophers’ ubiquity in AIREAD MORE “Netflix is struggling to get viewers to stick with its shows for more than a season.”Bloomberg’s Lucas Shaw, on Netflix’s second-season problemREAD MORE

Media News

Once Unimaginable, Publishers Are Preparing to Opt Out of Google Search

The nuclear option is gaining traction as web traffic collapses

UNLOCK FULL ACCESS

Mark Stenberg

Media

The 2026 Political Ad Outlay Was the Dumbest Money Spent In the History of Advertising

Democrats Learned to Win the Social Feed After 2024's Digital Drubbing

Canva’s Pricing Puzzle Is Every Marketer’s Problem

ADWEEK Tech Advantage: Brands Are Showing Up in ChatGPT. Proving It Pays Is Harder.

Anthropic Strikes Its First Licensing Deal, With the AP, for Election Coverage

Pastor Talarico, 3 Billion Views, and $56 Million on Broadcast: How James Talarico's Team Flipped Texas

Martech Entrepreneur Adam Robinson Among Nine Dead in Florida Plane Crash

Darshan Kantak Joins Netflix as VP of Ads Platform

AI Ad-Placement Companies Rembrand and Whisper World Merge, Eye IPO