Sometime in the next few months, someone at your company is going to paste two lines into robots.txt and feel responsible about it. Probably a developer acting on a legal memo. Maybe a CMS plugin with a friendly toggle. Possibly a CDN setting someone ticked in a security review.
The two lines will say your website does not want to be used for AI training. And depending on which AI company reads them, they will either do exactly that or quietly take your pages out of the answers your buyers are already reading.
That is the part of Friday’s announcement nobody put in the headline. It belongs on a marketing agenda, not just a legal one.
What happened on Friday
On October 30, Google, Microsoft and OpenAI each published posts saying they will honor one shared signal in robots.txt that tells participating AI companies not to train generative models on a site’s content. The signal comes from an IETF Internet-Draft written by Kevin Kelley of Anthropic and posted the same day. The Verge and PPC Land both report that Anthropic is building support as well. PPC Land quotes Kelley telling the working group mailing list that Anthropic “has already started implementing support for this draft.”
The mechanism is deliberately boring. Instead of naming a crawler on the User-Agent line, you name a “policy token”:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example, straight from the draft, says no training on anything except the press kit. Paths, longest match and grouping all work the way robots.txt already works.
Nobody reads it yet. Microsoft starts honoring it on February 1, 2027. Google’s post says 2027, and its crawler documentation now says early 2027. OpenAI says “the coming months.” Anthropic has given no date. The draft itself is an individual submission with no formal standing at the IETF, and it expires in May 2027 unless revised.
So nothing breaks this weekend. What changes is that a decision most brands never made on purpose now has a single, easy switch. Easy switches get flipped.
Same line, different meaning at Google
Here is the detail that matters for anyone who cares about showing up in AI answers.
Google says that when a site opts out of training with the shared token and does not mention Google-Extended, it plans to treat that as if the site had disallowed Google-Extended. Fine, except Google-Extended was never a training-only control. Google’s own crawler documentation says Google-Extended governs whether content can be used for training future Gemini models and “for grounding,” meaning content fed to the model at prompt time in Gemini Apps and in Grounding with Google Search on Vertex AI.
Google’s post spells out the other side of that trade: “Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.”
Put those together and the shared opt-out, at Google, also reaches past training into grounding. A site that writes the training block and nothing else should expect, once Google turns this on, to lose its eligibility to be cited in Gemini answers. PPC Land read the documents the same way. Google’s post does not walk through that case, so treat it as the plain reading of Google’s own words rather than a Google statement.
Microsoft went the opposite direction. It split training and grounding into separate preferences. The shared robots.txt token covers training only. Grounding is a separate, page-level setting with three values: Allow, Disallow, and a new DisallowExtended that keeps your pages in Microsoft’s own AI answers (Bing, Copilot, Microsoft 365) while keeping them out of the grounding Microsoft sells to other companies. Both default to Allow. Microsoft is blunt about the cost: choosing Disallow or DisallowExtended for grounding “will significantly reduce your content’s visibility in AI-generated experiences.” It adds a line every CMO should frame: “Neither choice guarantees inclusion, traffic, or payment.”
So the same two lines of text buy you two different outcomes. At Microsoft, no training, still cited. At Google, no training, and likely no Gemini citations either, unless you also write an explicit Google-Extended allow. Google’s plan only applies the translation when Google-Extended is not explicitly specified, so a named Google-Extended group still speaks for itself.
None of this touches your rankings. Microsoft says the preferences do not change eligibility for Bing results, and Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode sit under Search’s own controls, not this token. The exposure is in the assistant layer: the answers where your brand gets quoted, linked or left out.
Why this lands on the marketing desk
Legal will see “opt out of AI training” and hear “protect our IP.” That instinct makes sense for a newsroom or a research firm whose content is the product. For most brands, the website exists to be read, quoted and repeated. Your pricing page, product specs, comparison pages, help docs and press releases are marketing. You want models to know them, and you want assistants to cite them with a link.
Training and grounding do different jobs for a brand. Training shapes what a model believes about your category and your company before anyone asks. Grounding decides whether your page is the source behind the answer someone reads today, with a link back to you. Blocking the first is a long-term bet that your absence costs less than your presence. Blocking the second is a short-term visibility cut you will feel in referral data.
Very few brands should block both by accident. Some should block training on specific paths. Almost none should let a one-line default make that call for the whole domain.
Four ways this goes wrong quietly
The blanket block that says nothing. The draft states that wildcard groups, the familiar User-Agent: *, “MUST NOT be interpreted as matching the policy token.” If a file never names __usage_ai-training, no preference has been expressed. A staging-era “block everything” rule, or a security team’s catch-all, does not count as a training opt-out under this scheme. If you believed you had already said no, you have said nothing.
The group that eats a crawler. Under robots.txt rules, consecutive User-Agent lines form one group and share the rules beneath them, and the draft explicitly allows policy tokens and crawler names to sit in the same group. Paste the new token directly above an existing crawler line and the Disallow beneath it now applies to both. If that crawler is Bingbot or Googlebot, you did not opt out of training. You opted out of search. Have someone read the finished file group by group before it ships.
The old tag that stops working. Microsoft is retiring NoArchive and NoCache as AI controls. After March 1, 2027 they will no longer be read as training or grounding preferences, and Microsoft says that once it detects any of the new preferences on a site, it will disregard leftover NoArchive and NoCache tags there. A site that used NoCache to stay out of Copilot answers and then adds only the shared training token could lose that grounding restriction. Sites using those tags as of December 15, 2026 get a temporary grounding setting in Bing Webmaster Tools. Miss it, and those pages default to Allow until Microsoft recrawls them.
The vendors that are not in the room. The commitments cover Google, Microsoft, OpenAI and Anthropic. Meta’s communications head told The Verge the company supports a common signal but did not say Meta would honor it. Apple and Amazon made no commitment in any of the announcements. Your existing per-crawler rules still matter, and the draft says a private licensing deal can override the shared signal entirely.
Decide by page, not by domain
The useful exercise is a short map, not a policy memo. Sort your site into three buckets.
The first bucket is everything you want models to know and assistants to cite: product, pricing, specs, comparisons, docs, store locators, press. Leave training and grounding open. If someone insists on a sitewide training block, this bucket gets explicit Allow paths under the shared token and an explicit Google-Extended allow, so Gemini can still cite it.
The second bucket is content you sell or that sets you apart: original research, paid reports, proprietary benchmarks, course material. Here a training block can be a real decision. At Microsoft you can block training and keep grounding, so the work still earns citations. At Google, under the plan as written, you cannot split them for Gemini. Pick the trade on purpose and write down why.
The third bucket is the stuff nobody should be quoting anyway: internal search results, thin tag pages, expired campaigns, user-generated junk. Those are a crawl and indexing problem, so handle them with normal crawler rules rather than an AI preference.
Then name an owner. In most companies robots.txt belongs to whoever touched it last. Content-usage preferences are a brand visibility decision with legal input, so they need a marketing owner who signs off on changes the same way someone signs off on paid search budgets.
What to do this week
Pull your robots.txt and read it like a contract. Note every AI-related group you already have (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and the rest), when each was added and by whom. Check your CDN and CMS for AI-blocking toggles that write to that file without anyone opening it.
Ask legal one question in writing. Does the company want to block AI training across the whole site, on specific paths, or not at all? Attach the Google detail so the answer covers grounding, not just training.
Draft your bucket map. One page, three buckets, with example URLs. It takes an afternoon and saves you from a sitewide default chosen in a ticket.
Check for NoArchive and NoCache. If your site uses them, put December 15 and March 1 on the calendar and assign the Bing Webmaster Tools step to a person.
Baseline your AI visibility now. Record which assistants cite your key pages today, plus your Bing Webmaster Tools AI citation numbers if you have them. If someone flips the switch in February, you will want a before picture.
Write the change rule. Any edit to AI-related robots.txt groups, AI meta tags or headers goes through marketing review, and the finished file gets read group by group before deploy.
The shared token is a good idea. One line beats a growing list of vendor names, and publishers who never wanted to feed models finally get a clean way to say so. Just remember that the line is not neutral. At one company it means “don’t learn from me.” At another it also means “don’t quote me.” Make sure whoever types it knows which one they are paying for.
