For three years now, to tell AI company make e no use your website train mean say you sabi the name of im crawler. GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, meta tag for Microsoft. Every vendor get im own switch, im own syntax, im own documentation, and any website operator wey wan comot for AI training gats find dem all, one by one. Friday, the four biggest AI companies agree say dem go read one line instead.
OpenAI, Google, Microsoft and Anthropic each don announce say dem go respect one shared signal inside robots.txt, the small text file wey since 1994 don dey decide how machines go treat websites. One publisher fit write one rule wey dey talk to all the AI providers wey join at once, wey dey talk whether dem fit use im content train generative models. OpenAI talk say e go implement the signal for coming months. Google dey plan support for 2027. Microsoft publish the most detailed timeline, and new controls go start work for February.
The mechanism deliberately boring, and na im be the point. The proposal no add any new directive to robots.txt. Site dey write ordinary rule group under one policy token, __usage_ai-training, dey use the same Allow and Disallow lines wey people don dey use since. Wildcard group no fit ever match that token, and file wey no get token no dey show any preference — na so every robots.txt wey dem don publish for web today be. Nobody existing configuration no go change meaning overnight.
The idea come from the AI Preferences working group of Internet Engineering Task Force, and na Anthropic engineer Kevin Kelley submit the specific draft. Na individual Internet Draft, no be agreed standard, and e go expire May 2027 unless working group push am forward. Anthropic talk say to implement support across im services go take time. Na voluntary pact between competitors, no be law, and nothing dey inside beyond reputation.
Na for the fine print the agreement stop to look simple. Every company go read the same line and do different thing with am. Google plan to treat shared training opt-out like Google-Extended block, wey dey comot content from future Gemini training and from grounding inside Gemini Apps, but e no touch Google Search. Microsoft dey split the decision two: preference for training, preference for grounding, meaning whether your pages fit feed the answers wey Copilot and Bing AI Mode dey generate. Microsoft also dey retire the NoArchive and NoCache tags wey plenty sites dey rely on today, dey give operators till March 2027 to re-declare their preferences. Sites wey miss those deadlines go allow grounding by default.
That last detail deserve board attention. To comot for training no be the same thing as to comot for answers. Company fit keep im content out of model weights but still allow AI products to quote am, or the reverse, and every vendor dey draw the line for different place. Microsoft dey warn plainly: to restrict grounding go limit brand visibility for AI-generated answers, though no be for ordinary search results. This decision no longer be technical maintenance for web team. Na distribution strategy question about where your words suppose dey work hardest for you.
E still get honest limit to wetin change Friday. The signal na voluntary for everybody, including the companies wey announce am. E no talk anything about payment, licensing or attribution, and e no do anything for content wey don already enter today models. Publisher wey wan compensation still need contract; wey wan deletion still need lawyer. Wetin the shared line dey offer na consent machinery, cheap enough for any website to use and uniform enough say industry no fit claim say to comply too hard.
The deeper change deserve name. Training data dey shift from something the industry take to something e must ask for, and the asking dey standardized the way web dey standardize things: token, vocabulary, working group, expiry date. Once consent get file format, negotiation get starting position. Make una expect licensing talks, regulator drafts and enterprise data deals to reference this signal the way privacy debates dey reference cookie banners: as the moment wey vague principle turn operational control.
For leadership teams, the action small but real. Decide deliberately whether your content train models, feed answers, both or neither, and write that decision once, for one place, instead of make default answer for you. Appoint somebody to track how each provider dey interpret the shared line through 2026 and 2027, because the same words go mean different things at Google and Microsoft for a while. The companies wey treat am as strategic choice rather than settings page go be the ones wey their content work on their own terms.
Think am mathematically. Engineer am well-well. Lead with strategy. Build am responsibly. Scale am globally. Use artificial intelligence create real business value.
