OpenAI lets sites block their content from AI training via robots.txt
THE SHIFT
Website owners have long wondered if they can signal to AI services that their pages should stay out of training data. OpenAI now offers a way.
The company says it will respect a new robots.txt directive that lets publishers declare a preference against using their content for model training.
THE PROOF
OpenAI’s announcement explains that it will honor a shared robots.txt signal, treating it as a binding preference for participating AI providers.
The policy is built into the company’s data‑crawling pipelines, so any site that adds the directive will be excluded from future training sets.
THE CATCH
The signal only applies to providers that opt‑in, and OpenAI’s promise does not create legal enforcement; non‑participating services may still scrape.
Implementation details are sparse, and the directive does not retroactively remove data already used, leaving past training footprints untouched.
YOUR MOVE
If you run a website, add the line `OpenAI-Crawler: no‑train` (or the specified token) to your robots.txt today.
Developers should update their crawlers to check for the signal and respect it, while monitoring OpenAI’s compliance reports for updates.
OpenAI’s robots.txt signal gives sites a practical, though voluntary, tool to keep future AI training out of their pages.
Sources & further reading
- OpenAI News official RSS summary: Making it easier for websites to express AI training preferences ↗
Published by The Breakdown. How we report · Our source desk.
