Topics
Latest
AI
Amazon
Apps
Biotech & Health
Climate
Cloud Computing
Commerce
Crypto
Enterprise
EVs
Fintech
Fundraising
Gadgets
Gaming
Government & Policy
Hardware
Layoffs
Media & Entertainment
Meta
Microsoft
Privacy
Robotics
Security
Social
Space
Startups
TikTok
Transportation
Venture
More from TechCrunch
Staff
Events
Startup Battlefield
StrictlyVC
Newsletters
Podcasts
Videos
Partner Content
TechCrunch Brand Studio
Crunchboard
Contact Us
Patreon stops asking AI bots not to scrape — and starts blocking them
Patreon, the membership platform for creators, is cracking down on AI scraping its content for training purposes. On Thursday, the company shared that it’s working with internet infrastructure provider Cloudflare to directly block access to AI bots designed to train their AI models on creators’ work without permission.
The strengthened measures were necessary because AI scraping has become more sophisticated since it first put measures in place to deter AI crawlers in 2023, the company says. In addition, Patreon’s paywall has long locked much of creators’ content out of reach of crawlers. But more recently, the company introduced new discovery tools like a redesigned Home Feed and its tweet-like Quips, which could expose more content to crawlers.
The changes come about as more online publishers and content creators are coming to grips with how AI is ingesting their work for the purpose of making their AI models smarter. To combat this, Cloudflare now offers tools that allow website publishers to restrict AI bots, including a marketplace that lets websites charge AI bots for scraping, dubbed Pay Per Crawl. Earlier this month, it changed its policies so that “mixed-use” crawlers, meaning those that both index and train on a website’s content, are blocked by default on any pages that host ads.
marketplace that lets websites charge AI bots for scraping
Patreon says that it’s extending its existing work with Cloudflare to use the company’s AI Crawl Control technology to update its AI policies and enforcement tools. The difference here is that instead of simply asking AI crawlers not to scrape content using the robots.txt files — a standard way to provide bots with instructions on how they can use its site — Patreon is now actively blocking AI training bots.
“Consent shouldn’t depend on whether a scraper chooses to behave,” a Patreon blog post explains, referencing the stricter measures.
When testing the features, individual AI training crawlers’ weekly attempts to access Patreon went from “thousands of attempts to zero,” the post noted. That indicates that the AI scrapers were ignoring Patreon’s robots.txt file and scraping the site anyway, despite its requests.
However, the company said that it will allow bots that index pages and organize information that can be used to send users back to Patreon.
“As AI agents become increasingly powerful and popular, creators deserve a meaningful say in how their work is used by AI companies,” remarked Patreon’s product chief Drew Rowny in the announcement. “On most of the Internet, creators have to accept AI training on their work just to reach and grow an audience. Patreon has a different vision: creators should be able to grow their audience and control how their work is used.”
Topics
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.
we may earn a small commission
Consumer News Editor
Last chance to save up to $190 on TechCrunch Founder Summit. Join 1,000+ founders and VCs at all stages for real-world scaling insights and connections that move the needle.Savings end June 26, 11:59 p.m. PT.
Most Popular
Tesla driver in fatal Texas crash pressed accelerator 100%, NTSB confirms
Sean O'Kane
Tesla driver in fatal Texas crash pressed accelerator 100%, NTSB confirms
Tesla driver in fatal Texas crash pressed accelerator 100%, NTSB confirms
Tesla driver in fatal Texas crash pressed accelerator 100%, NTSB confirms
Sean O'Kane
Amid hardware legal battle, OpenAI releases a $230 keyboard for Codex
Lucas Ropek
Amid hardware legal battle, OpenAI releases a $230 keyboard for Codex
Amid hardware legal battle, OpenAI releases a $230 keyboard for Codex
Amid hardware legal battle, OpenAI releases a $230 keyboard for Codex
Lucas Ropek
OpenAI’s first hardware device is reportedly a screenless speaker that can move
Lucas Ropek
OpenAI’s first hardware device is reportedly a screenless speaker that can move
OpenAI’s first hardware device is reportedly a screenless speaker that can move
OpenAI’s first hardware device is reportedly a screenless speaker that can move
Lucas Ropek
Anthropic’s newest ad is creeping people out
Lucas Ropek
Anthropic’s newest ad is creeping people out
Anthropic’s newest ad is creeping people out
Anthropic’s newest ad is creeping people out
Lucas Ropek
Satya Nadella has issued a shocking warning to companies using AI
Julie Bort
Satya Nadella has issued a shocking warning to companies using AI
Satya Nadella has issued a shocking warning to companies using AI
Satya Nadella has issued a shocking warning to companies using AI
Julie Bort
The wildest allegations in Apple’s trade secrets lawsuit against OpenAI
Sarah Perez
The wildest allegations in Apple’s trade secrets lawsuit against OpenAI
The wildest allegations in Apple’s trade secrets lawsuit against OpenAI
The wildest allegations in Apple’s trade secrets lawsuit against OpenAI
Sarah Perez
Anthropic starts localizing Claude pricing for India, its biggest market after the US
Jagmeet Singh
Anthropic starts localizing Claude pricing for India, its biggest market after the US
Anthropic starts localizing Claude pricing for India, its biggest market after the US
Anthropic starts localizing Claude pricing for India, its biggest market after the US
Jagmeet Singh