Do you need AI SEO files? A platform-by-platform checklist

Audit the existing page before adding a new file

Choose one important URL and inspect its response, visible content, canonical and indexing directives. Confirm that ordinary internal links lead to it. Check whether the relevant crawler can retrieve the page and the resources needed to understand it. A beautifully formatted AI file cannot repair an unavailable page or an accidental noindex.

Next, write the specific problem in one sentence: discovery in a named search product, a publisher participation preference, training collection or a documentation agent finding the right reference. That sentence determines which control to investigate.

Google Search: separate Search eligibility from other AI products

Google's current guidance says special AI text files, llms.txt and special schema markup are not required for its generative Search features. Ordinary Search foundations still matter. Search Console also documents a Search generative AI participation control; inspect the applicable property setting when eligibility is the question.

Google-Extended is a different control. Google's crawler documentation states that it does not determine inclusion or ranking in Google Search. Do not use it as a substitute for reviewing Googlebot access, indexing directives or the Search-specific participation setting.

The practical order is page accessibility, indexing eligibility, the relevant product setting, then content quality. When reviewing a supplier's recommendation, ask which documented Google mechanism it changes. A generic AI-readiness score is not an explanation of that mechanism.

ChatGPT, Claude and Perplexity have different crawler roles

OpenAI documents OAI-SearchBot for search and GPTBot for potential model-training collection. Those preferences are independent. ChatGPT-User handles some user-directed requests and is not the automatic search crawler; OpenAI notes that robots.txt rules may not apply to those user-initiated actions.

Anthropic distinguishes Claude-SearchBot, ClaudeBot and Claude-User. Perplexity describes PerplexityBot as its search crawler, not a foundation-model training crawler. Apply each vendor's current definitions rather than copying one provider's user-agent names into a universal allowlist.

Where vendors publish IP-verification guidance, use it when diagnosing requests. A user-agent string is useful context, but it is not, by itself, a reliable identity check. Keep any access exception narrow and tied to the documented service.

Use llms.txt when it has a real consumer

The llms.txt project is a proposal for agent-friendly website context and links, including links to clean Markdown documentation. Its current version is useful to evaluate for documentation workflows and tools that support it. Publication of such a file is different from a search engine using it as a ranking input.

Before maintaining one, identify the consuming workflow, the pages it needs and the person responsible for updates. For a software documentation site, a short index of versioned API references may save an agent several navigation steps. For a small local-service site, clear public service pages may solve the immediate discovery problem more directly.

If you publish the file, keep links canonical and current. Remove obsolete endpoints, distinguish stable references from temporary examples and avoid placing private instructions or credentials in a public document. Measure whether the intended tool actually uses it instead of treating successful file creation as a visibility result.

Test one policy change at a time

Save the existing rules and identify the exact groups affected. Then test both the intended path and an unrelated path. Review response headers, robots directives and server evidence together. A local syntax check is useful, but it does not establish what a remote service has fetched or indexed.

This example expresses a specific OpenAI preference: allow automatic search discovery while opting out of GPTBot collection. It illustrates separate decisions; adapt paths and groups to the site's existing policy rather than replacing an entire robots.txt file blindly.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Record the deployed version and review it after the service has had time to process changes. Keep ranking or citation measurement in a separate record from the access-policy test. The AI-search measurement checklist provides that reporting structure.

A practical approval checklist

A useful implementation ticket names the consumer, the intended behaviour and a verification method. If those three items are missing, the proposed file is still an idea rather than a technical requirement.

• Name the product and the exact crawler or consuming agent.
• Separate search inclusion, training collection and user-directed access.
• Check existing page responses, indexing directives and internal links.
• Use the platform's current documented control.
• Preserve a rollback version and test affected paths.
• Confirm the intended behaviour, then measure visibility independently.