For three years, telling an AI company not to train on your website meant knowing its crawler's name. GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, a meta tag for Microsoft. Every vendor had its own switch, its own syntax and its own documentation, and any website operator who wanted out of AI training had to find and maintain all of them. On Friday, the four largest AI companies agreed to read a single line instead.
OpenAI, Google, Microsoft and Anthropic each announced that they will honor a shared signal in robots.txt, the small text file that has governed how machines treat websites since 1994. A publisher will be able to write one rule, addressed to every participating AI provider at once, stating whether its content may be used to train generative models. OpenAI says it will implement the signal in the coming months. Google plans support in 2027. Microsoft has published the most detailed timetable, with new controls taking effect from February.
The mechanics are deliberately boring, and that is the point. The proposal adds no new directive to robots.txt. A site writes an ordinary rule group under a policy token, __usage_ai-training, using the same Allow and Disallow lines it has always used. A wildcard group can never match the token, and a file without the token expresses no preference at all, which is true of every robots.txt file on the web today. Nobody's existing configuration changes meaning overnight.
The idea originates inside the Internet Engineering Task Force's AI Preferences working group, and the specific draft was submitted by Anthropic engineer Kevin Kelley. It is an individual Internet Draft, not an adopted standard, and it expires in May 2027 unless the working group takes it further. Anthropic says implementing support across its services will take time. This is a voluntary pact among competitors, not a regulation, and nothing in it carries a penalty beyond reputation.
The fine print is where the agreement stops looking simple. Each company will read the same line and do something different with it. Google plans to treat a shared opt-out as a Google-Extended block, which removes content from future Gemini training and from grounding inside Gemini Apps, but leaves Google Search untouched. Microsoft is splitting the decision in two: one preference for training, another for grounding, meaning whether your pages can inform the answers Copilot and Bing's AI Mode generate. Microsoft is also retiring the NoArchive and NoCache tags that many sites rely on today, giving operators until March 2027 to restate their choices. Sites that miss that deadline will default to allowing grounding.
That last detail deserves a board's attention. Opting out of training is not the same as opting out of answers. A company can keep its content out of a model's weights while still letting AI products quote it, or the reverse, and each vendor draws that line in a different place. Microsoft is blunt about the trade-off: restricting grounding will limit a brand's visibility in AI-generated answers, though not in ordinary search results. The decision is no longer technical housekeeping for the web team. It is a distribution strategy question about where your words work hardest for you.
There is also an honest limit to what changed on Friday. The signal is voluntary for everyone, including the companies that announced it. It says nothing about payment, licensing or attribution, and it does nothing for content already absorbed into today's models. A publisher that wants compensation still needs a contract; a publisher that wants erasure still needs a lawyer. What the shared line offers is consent machinery, cheap enough for any website to use and uniform enough that the industry cannot claim it was too complicated to respect.
The deeper shift is worth naming. Training data is moving from something the industry took to something it must ask for, and the asking is being standardised the way the web standardises things: a token, a vocabulary, a working group, a deadline. Once consent has a file format, negotiation has a starting point. Expect licensing talks, regulator drafts and enterprise data deals to reference this signal the way privacy discussions reference cookie banners, as the moment a vague principle became an operational control.
For leadership teams, the action is small but real. Decide deliberately whether your content trains models, informs answers, both or neither, and write that decision down once, in one place, rather than letting a default answer for you. Assign someone to track how each provider interprets the shared line through 2026 and 2027, because the same words will mean different things at Google and Microsoft for a while. The companies that treat this as a strategy choice rather than a settings page will be the ones whose content works on their own terms.
Think mathematically. Engineer intelligently. Lead strategically. Build responsibly. Scale globally. Create real business value through artificial intelligence.
