Google, OpenAI, Anthropic and Microsoft have committed to honoring a new robots.txt directive that lets publishers specify their preferences when it comes to the scraping of their content for generative AI training, Engadget reports.

The companies have adopted a proposal under consideration at the IETF, the open standards organization for the internet, written by Anthropic technical lead for standards Kevin Kelley.

Website owners will set up a new user agent in their robots.txt file specifically for AI training, for example blocking all scraping while allowing it in specific areas such as a press kit.

According to Engadget, the directive is interpreted as a stop sign for the Google-Extended user agent, which crawls websites for materials used to train Gemini models.

The change means publishers no longer have to identify individual crawlers and add separate rules for each provider — one common signal now communicates their preference to multiple participating AI companies at once.

thurrott.com reports that Microsoft framed the effort as giving publishers choices about training or grounding while remaining eligible for Bing search results, so sites can stay discoverable without having their work used for models.

But Thurrott also argues the move is largely symbolic because it comes years after these companies scraped content at a scale never before seen, and the directive cannot undo any past scraping the companies have already done.

The IETF working group will propose the system as an international standard once complete, according to thurrott.com.

Adoption is staggered: Anthropic has already started supporting the new directive, while Engadget reports that Microsoft said it will begin supporting it on February 1, 2027, Google said it will support it in early 2027 as well, and OpenAI pledged to implement the new signal in the coming months.

The opt-out is voluntary, meaning each participating company is committing to honor the signal rather than being required to by law.

Publishers who want to block AI training can now add the new user-agent section to their robots.txt file, with an Allow line to carve out specific parts of their site if they choose.

The directive blocks crawlers specifically meant to gather data for AI training only — it does not prevent crawling altogether or stop content from showing up in Google search, Engadget explains.

That separation is the core of the deal: websites can refuse AI training use of their work without sacrificing search visibility, ending what Microsoft's Krishna Madhavan described as the old choice between discoverability and control over how generative models use content.