Getting your business quoted by AI assistants
AI crawlers do different jobs and need different decisions: training crawlers like GPTBot build models, search crawlers like OAI-SearchBot build the index an assistant answers from, and user-triggered fetchers visit because somebody asked. Blocking the first does not remove you from the second, and conflating them is how businesses accidentally make themselves unquotable.
Three kinds of crawler, three different decisions
Treating “AI crawlers” as one category is the root of most bad decisions here. There are three jobs, and you may reasonably want different answers for each.
Training crawlers collect content to train models. GPTBot, ClaudeBot, CCBot,
Google-Extended and Applebot-Extended are in this group. Blocking them keeps your content out
of future model weights and has no effect on whether an assistant can find you today.
Search crawlers build the index an assistant searches when answering a live question.
OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot do this. These are the ones
that decide whether you can be cited at all. Blocking them is how a business makes itself
invisible to assistants while believing it has protected something.
User-triggered fetchers visit a page because a person asked an assistant to look at it.
ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User behave this way. Blocking
them means a potential customer who pastes your URL gets told the page cannot be read.
For a local business the answer is usually: allow all three. Your product is a service, not the prose on the page, and being quotable is the entire point.
A robots.txt that says what you mean
robots.txt groups do not inherit. A rule under User-agent: * is replaced, not supplemented, by
a more specific group - so a crawler you name needs its own complete rules.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlCrawler names change, and vendors add new ones without much announcement. Check the operator’s own documentation rather than trusting a list in a blog post - including this one.
The Google-Extended trap
This one is worth stating on its own because it is expensive and widely misunderstood.
Google-Extended controls whether your content may be used for Gemini training and grounding. It
does not control AI Overviews in Google Search. Those are served from the ordinary Search
index, crawled by Googlebot. Blocking Google-Extended therefore does not remove you from AI
Overviews - it only gives up the other use while leaving you exactly as exposed.
If you genuinely want out of AI Overviews, the lever is nosnippet or max-snippet, and those
also cut your normal search snippet. There is no free version of that choice.
Is llms.txt worth it?
Honestly: probably, barely, and not for the reasons it is usually sold.
llms.txt is a proposed convention, not a standard. No major AI company has publicly
committed to reading it, and there is no good public evidence that it changes what assistants say
about you. Agencies selling it as essential are ahead of the facts.
We ship one anyway, because it is half an hour of work, it costs nothing to serve, and the
downside is zero. Keep it generated from the same content as your pages so it cannot drift, and
serve it with X-Robots-Tag: noindex so it does not compete with the real pages as duplicate
content.
Write the answer, then the page
The single highest-leverage thing is unglamorous: put a direct, self-contained answer in the first
paragraph under each <h1>.
Assistants extract passages. A paragraph that answers the page’s question completely, in forty to seventy words, without depending on the heading above it or the paragraph below it, is a paragraph that can be lifted and attributed. A page that opens with “Welcome to our website” offers nothing to lift.
Every page on this site is built that way, including this one. Look at the paragraph under the heading at the top and you will see the pattern: it answers the question on its own, so it survives being quoted somewhere else.
Questions we get asked
Is llms.txt an official standard?
No. It is a proposed convention, not a specification any major AI company has committed to honouring, and you should treat claims about its impact with scepticism. We ship one because it is half an hour of work and costs nothing, not because there is evidence it changes outcomes. Anyone presenting it as essential is overstating what is known.
Should I block AI crawlers?
It depends what you sell. A publisher whose product is the writing has a real argument for blocking training crawlers. A local business whose product is a service wants to be found and quoted, and blocking the crawlers that feed assistant answers removes you from a channel where your competitors will still appear.
Does blocking Google-Extended remove me from AI Overviews?
No, and this is the most expensive misunderstanding in this area. Google-Extended governs whether your content is used for Gemini model training and grounding. AI Overviews in Search are served from the normal Search index via Googlebot, so blocking Google-Extended does not take you out of them - it only costs you the other use.
Tell us what you need built.
A short call, a straight answer on scope and cost, and a fixed quote. No retainer, no monthly fee, no commission on your orders.