Updated 4 minutes ago

Share this article

Related News

Anthropic Pays $3.4 Billion for 1.7GW of Gas Generators to Power AI Data Centers

Nov 9, 2026

Anthropic Pays $3.4 Billion for 1.7GW of Gas Generators to Power AI Data Centers

Anthropic has spent $3.4 billion to take over a queue position for 1.7 gigawatts of Caterpillar gas generators originally ordered by OnePWR Solutions. It is the largest known single purchase of power-generation equipment by an AI lab, as demand outruns factory capacity for new generator orders.

anthropicclaudeai-infrastructure

OpenAI Acquires Pierre to Build End-to-End Dev Platform in Codex

Nov 9, 2026

OpenAI Acquires Pierre to Build End-to-End Dev Platform in Codex

OpenAI is acquiring Pierre Computer Company, maker of Code Storage and open-source Git tools Trees and Diffs, to transform Codex from an AI coding assistant into a full-spectrum development platform. The deal brings Pierre's source control, code review, and project organization tools into Codex, building on OpenAI's recently launched Pando Git backend.

openaicodexpierre-computer

Claude Code's 'Don't Ask Again' Exposed 2,539 Secrets on GitHub

Nov 9, 2026

Claude Code's 'Don't Ask Again' Exposed 2,539 Secrets on GitHub

Security firm Apiiro found that Claude Code's "don't ask again" feature stores passwords, API keys, and SSH logins in plain text in a local settings file — and over 1,000 copies ended up in public GitHub repositories. Anthropic's Claude Code lead acknowledged the issue but existing fixes may not fully solve it.

claude-codeanthropicsecurity

OpenAI, Google, Microsoft, and Anthropic Back Unified AI Training Opt-Out in Robots.txt

AI Training Control

OpenAI, Google, Microsoft, and Anthropic Back Unified AI Training Opt-Out in Robots.txt

By OpenTools TeamNovember 9, 2026

The four largest Western AI labs will honor a single robots.txt policy token that lets website owners opt out of AI training across all their crawlers at once. Anthropic has already begun implementation; OpenAI, Google, and Microsoft follow in 2027.

One Token to Block Them All

OpenAI, Google, Microsoft, and Anthropic announced on October 30 that they will honor a single shared signal in robots.txt that lets website owners opt their content out of AI model training across all four companies' crawlers at once, The Verge reported. The policy token — __usage_ai-training — is proposed as an IETF Internet-Draft by Kevin Kelley, a technical lead for standards at Anthropic.

Today, a site that wants to block AI training crawlers must write separate rules for each company's bot: GPTBot for OpenAI, Google-Extended for Google, Claude-Web for Anthropic, and so on. The new standard collapses that into a single line: User-Agent: __usage_ai-training followed by Disallow: /, The Next Web explained.

Staggered Rollout Through 2027

Implementation timelines vary significantly. Anthropic says it has already started implementing support, Kelley wrote on the IETF mailing list. OpenAI says it aims to implement the signal "in the coming months." Microsoft targets February 1, 2027, and will replace the NoArchive and NoCache tags it has used since 2023. Google says 2027, The Next Web noted.

Google Search Advocate John Mueller wrote that if a site opts out using the token without mentioning Google-Extended, Google will treat it as if the publisher had disallowed Google-Extended directly, AI Weekly reported. Sites that allow Google-Extended can still appear in Gemini app answers and receive search traffic.

What the Signal Does — and Doesn't — Do

The token is a preference signal, not a crawling directive. As the IETF draft states, "this policy token is not a directive for crawling behavior" — bots may still fetch pages even when training is disallowed, AI Weekly noted. As AI Weekly characterized it, OpenAI frames the mechanism as "a coordination mechanism rather than a security control" under RFC 9309, meaning non-compliant scrapers can disregard it entirely.

There are other hard limits. Updating robots.txt does not remove material already collected into existing training datasets or trained models. The draft explicitly notes it is an individual submission, not endorsed by the IETF, and has no formal standing in the standards process. Private agreements between a site and an AI company can override it. Microsoft added that the new preferences "do not establish licensing or payment arrangements," per The Next Web.

Microsoft Adds Grounding Toggle, Meta Signals Support

Microsoft is layering on additional controls. Beyond the shared training opt-out, the company is adding a separate page-level setting for AI grounding — the retrieval step that pulls live web content into chat answers rather than training model weights. One option allows Microsoft's own products such as Copilot; it blocks the grounding services Microsoft sells to other companies, Microsoft wrote in its announcement.

Meta communications head Andy Stone voiced support for a common signal on X, The Verge reported, but did not say whether Meta would honor it. "We believe creating a common signal that conveys such preferences to multiple providers at once is an important step," Stone said. Participants listed in the IETF draft include Cloudflare, the Common Crawl Foundation, News Corp, the Financial Times, the BBC, the Electronic Frontier Foundation, and Creative Commons.

Why This Matters for Site Owners and Builders

For developers and site owners, the practical change is simplicity. Instead of maintaining per-crawler rules for a growing list of AI bots, a single line expresses a training preference across the four companies that collectively dominate AI model development. Google noted the proposal is consistent with the EU's Code of Practice for general-purpose AI under the AI Act, which recognizes robots.txt as a primary machine-readable preference mechanism, according to The Next Web.

The token will also appear in the context of ongoing publisher demands for payment. In July, Cloudflare told AI crawlers to pay or get blocked. Its CEO has warned that bot traffic could reach 1,000 times human web traffic. The shared opt-out standard doesn't solve the payment question, but it does give every site operator — from indie bloggers to major publishers — a single, clear mechanism to say no to AI training across the industry's largest players.

Tags

robots-txtai-trainingweb-crawlersopenai google microsoft anthropic data-privacyweb-standardsietf