# How to optimize your website for AI search: what is proven and what is guesswork

> An AI answer is assembled from several sub-searches and from pages a crawler could actually fetch and read. Most sites that go uncited fail at the fetch, through a firewall rule, a robots.txt group or JavaScript-only content, not through weak writing. Fix access first, then write one self-contained answer per sub-question, and treat tactics beyond that as hypotheses.

By LAIN, 23 Oct 2026. 11 min read. Category: Website. Canonical: https://lainagency.com/resources/optimize-website-for-ai-search

Most advice on AI search is about what to write. The sites that never get cited usually lose earlier, at the fetch: a firewall that turns bots away, a robots.txt rule that does not do what its author thinks, or an answer that only exists after JavaScript runs. Those failures are mechanical, they take an hour to check, and nothing you write can fix them.

This guide goes in that order. How an answer is assembled, then the five access failures, then how to write for the searches you cannot see, then an honest look at which claims have evidence.

## How an AI answer is assembled
Three facts shape everything else.

**One question becomes several searches.** Google describes AI Mode as issuing multiple related searches at the same time across subtopics, a technique it calls query fan-out. Microsoft shows the same thing from the publisher side: the AI Performance report in Bing Webmaster Tools (public preview since February 2026) lists *grounding queries*, the shorter queries the system generated in the background to find the pages it cited. So you are retrieved for questions the user never typed. A page that answers one narrow sub-question well can be cited for a broad question it does not mention in its title.

**Three different jobs hide behind "AI crawler".** OpenAI and Anthropic each publish separate names for training, for search indexing and for fetching a page because a person asked. OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User as independent controls. Anthropic does the same with ClaudeBot, Claude-SearchBot and Claude-User. Allowing one says nothing about the others.

**Being cited is not the same as being visited.** In Pew Research's analysis of the browsing of 900 US adults (68,879 Google searches, March 2025), people clicked a regular result in 8% of visits that showed an AI summary and 15% of visits that did not. A link inside the summary itself was clicked in about 1% of visits. Plan for visibility and brand recall first, and treat clicks as a bonus. Pew's panel is American and a year old, so check your own analytics.

## Five access failures, in the order to check them
These decide whether anything else you do is read.

| Failure | Why it happens | How to detect it | Fix |
|---|---|---|---|
| The firewall says no | Cloudflare made blocking known AI crawlers the default for new domains from July 2025, and security plugins and hosts have similar switches | Bot analytics or server logs showing 403 or challenge responses for OAI-SearchBot, Claude-SearchBot, PerplexityBot | Allow the search crawlers you want in the bot settings, by verified bot name |
| The answer needs JavaScript | Vercel and MERJ analysed AI crawler traffic across Vercel's network in late 2024 and found none of the major AI crawlers rendered JavaScript. They fetch script files but do not run them. Googlebot does render | Fetch the page without a browser and search the raw HTML for your answer sentence | Server-render or pre-render the pages that should be cited |
| A robots.txt group replaces `*` | A crawler obeys the group that names it and ignores `*` (see the worked example below) | Test each bot name against your real paths | Repeat the shared rules in every named group |
| The answer sits too deep | Google's crawler documentation says Googlebot fetches the first 2 MB of an HTML file for Search. HTTP Archive puts the median HTML page at about 33 KB, but inline bundles and page builders can exceed 2 MB, and reports differ on whether Google changed behaviour or only the wording | Check the size of the raw HTML | Keep the answer high in the markup and scripts and styles external |
| A change has not landed yet | OpenAI says search-related robots.txt changes can take about 24 hours to take effect | Wait, then retest | Do not judge a change on the same day |

To check the second row, fetch the page the way a crawler does and look for a sentence that should be there: `curl -s https://yoursite.com/page | grep -c "your answer sentence"`. A result of 0 means the sentence is added by script. Google's own crawler may still index it. The others, on the evidence above, will not see it.

### Worked example: the robots.txt group trap
Here is a file written by someone who wants to block one training crawler and welcome one search crawler, while keeping two private folders out of everything.

- `User-agent: *` with `Disallow: /account/` and `Disallow: /staging/`
- `User-agent: GPTBot` with `Disallow: /`
- `User-agent: OAI-SearchBot` with `Allow: /`

We ran it through Python's `urllib.robotparser`. The result for OAI-SearchBot: `/`, `/blog/a`, `/account/orders` and `/staging/x` are all fetchable. The `*` rules do not apply to it because it has a group of its own. RFC 9309 says a crawler uses the group that matches its product token, and falls back to `*` only if none does. Groups for different bots are never merged. The fix is to repeat the two `Disallow` lines inside the `OAI-SearchBot` group, after which the same test blocks both folders for it and still allows the rest. Googlebot and ClaudeBot, which have no group, still obey `*`.

Two limits to know. robots.txt is a request, not a lock: Cloudflare reported in August 2025 that Perplexity used undeclared crawlers to reach sites that had blocked its declared bots, and Perplexity called the report misleading. Cloudflare's test and Perplexity's account conflict, so if content is genuinely private, protect it with authentication. And a private page should also not be in your sitemap.

## Decide which crawlers you want
The decision is about purpose, not about the company.

| Your goal | OpenAI | Anthropic | Google |
|---|---|---|---|
| Appear in answers and be fetchable when a person asks | Allow OAI-SearchBot. ChatGPT-User is user-initiated and OpenAI says robots.txt rules may not apply to it | Allow Claude-SearchBot and Claude-User | Allow Googlebot. There is no separate AI crawler for AI Overviews |
| Stay out of model training, stay in answers | Disallow GPTBot only | Disallow ClaudeBot only | Disallow Google-Extended, which does not affect Search inclusion or ranking |
| Stay out of Google's AI answers but keep ordinary snippets | Not applicable | Not applicable | Use the Search generative AI control in Search Console, if it is available to your account |

On the last row, `nosnippet`, `data-nosnippet` and `max-snippet` also limit AI features, but they remove ordinary snippets too. Google introduced a property-level control in Search Console that leaves normal snippets alone. Reports disagree on how widely it has rolled out, so look in your own Settings.

For reference, lainagency.com allows every crawler named above in its robots.txt and ships complete static HTML for each page, so crawlers that do not run scripts still read the content.

## Write for the searches you cannot see
If one question fans out into several searches, the unit to plan is the sub-question, not the keyword. A method that works with any topic:

1. Write the head question a customer would ask. Example: "How much does a website redesign involve for a 40 page site?"
2. Ask what a careful person would need to look up before answering. Typical sub-questions: what changes in the process for a site that size, what must be redirected, how long does it take, what can go wrong, what should I prepare.
3. Give each sub-question one section whose first sentence is the answer, in a form that still makes sense lifted out of the page. "Redirect every old URL that has links or traffic to its closest new page with a single 301" survives extraction. "As mentioned above, this is important" does not.
4. Put the conditions in the answer. "6 to 10 weeks for a typical business site" is quotable. "It depends" is not.
5. Link the sub-answers to each other so a crawler that lands on one can reach the rest.

Be honest about the limit of this method. Google has described fan-out; it has not published how it picks sub-queries, so you are inferring them. Use Bing's grounding queries and your Search Console query data to check your guesses against what was actually retrieved.

## What the evidence supports
There is less proof here than the industry implies.

- **Adding evidence to a page helped in one academic benchmark.** The GEO paper (Aggarwal and colleagues, KDD 2024) tested several content tactics on a benchmark of queries and reported visibility gains of up to 40%, and up to 37% on Perplexity. The strongest were adding statistics, adding quotations and citing sources. Keyword stuffing, the classic SEO move, scored below the unmodified baseline. Effects varied by topic. It used 2023 and 2024 models, so treat it as a direction, not a promise.
- **Ranking well and being cited overlap less than you would hope, and nobody agrees by how much.** Studies of how many AI Overview citations also sit in the organic top 10 range from about 17% (BrightEdge, earlier) through 32% (seoClarity) and 38% (Ahrefs, down from 76% in its first version) to more than half in other counts. Differences in query samples and in what counts as a citation explain much of the spread. The honest reading is that top 10 rankings help but do not decide it.
- **AI answers are unstable.** SparkToro and Gumshoe had 600 volunteers run 12 recommendation prompts 2,961 times across ChatGPT, Claude and Google's AI. The same prompt returned the same brand list less than once in 100 runs. The study is not peer reviewed. Its lesson holds anyway: one screenshot of an answer proves nothing.
- **Structured data: required to be honest, unproven as a lever.** Google's guidelines say not to mark up content readers cannot see. Claims that schema raises citation rates come from vendors and conference remarks that we could not trace to a primary source or controlled test. Use it because it describes the page accurately.
- **llms.txt: no confirmed use.** Google's John Mueller said that, as far as he knows, no AI service has said it uses the file, and compared it to the keywords meta tag. Search Engine Land argues the comparison was narrower than headlines suggest. Either way, no major provider has announced support. It costs little. Do it last.

## Measure without fooling yourself
1. **Search Console.** AI Overview and AI Mode traffic is counted in the normal Web search totals. Since June 2026 Google has also been testing generative AI performance reports that show impressions from these features, in stages.
2. **Bing Webmaster Tools.** The AI Performance report shows cited pages and grounding queries for Copilot and Bing's AI summaries. It is sampled and has no click data.
3. **Server logs.** Count requests by user agent and status code per week. A bot that never appears, or only receives 403s, is your answer.
4. **A repeatable prompt sample.** Pick 15 real customer questions, run each three times in each assistant you care about, and record the share of runs in which you are named or cited. Because lists vary run to run, report a rate, never a single result.

## Where this goes wrong
- **The CDN blocks the bots and nobody knows.** Cause: a security default applied when the domain was added. Detect it in the bot logs. Fix it in the firewall, not in robots.txt.
- **The page is a shell.** Cause: a front end that builds the text in the browser. Detect it with the raw HTML test. Fix it with server rendering.
- **Structured data describes a page that is not there.** Cause: markup written to chase a feature. Fix: generate it from the same content as the visible page, which is how every LAIN page is built.
- **Judging a change in a day.** Cause: crawl and index delays. Fix: wait a week and use the prompt sample, not one query.
- **Blocking search crawlers to stop training.** Cause: treating all AI bots as one thing. Fix: use the purpose table above.

## A pass or fail checklist
- The raw HTML of each key page contains its main answer sentence.
- Every named group in robots.txt repeats the private path rules.
- Logs show 200 responses to the search crawlers you want.
- Each section starts with an answer that survives being quoted alone.
- Structured data matches visible text and passes the Rich Results Test.
- You can state, with numbers, how often you appear in a fixed set of prompts.

If you want these checks built into a [website project](https://lainagency.com/about#website-design) or run on an existing site, [send us the link](https://lainagency.com/contact). Related reading: the [redesign SEO checklist](https://lainagency.com/resources/website-redesign-seo-checklist) covers the redirects and indexing side, and the [accessibility checklist](https://lainagency.com/resources/website-accessibility-checklist) overlaps heavily, since real text and clear structure serve both.

## Frequently asked questions

### Do AI crawlers run JavaScript?

In Vercel and MERJ's analysis of AI crawler traffic, none of the major AI crawlers rendered JavaScript, although they downloaded script files. Googlebot renders pages. If your key text is added by script, test whether it appears in the raw HTML.

### Will blocking GPTBot remove me from ChatGPT answers?

No. OpenAI documents GPTBot (training), OAI-SearchBot (search results) and ChatGPT-User (user-initiated fetches) as independent controls. To appear in ChatGPT search you allow OAI-SearchBot. Robots.txt changes can take about 24 hours to take effect.

### Why does my new robots.txt rule for one bot ignore my other rules?

Under RFC 9309 a crawler follows the group that names it and does not combine it with the * group. If you add a group for OAI-SearchBot, repeat any Disallow lines from the * group inside it, or the bot may fetch those paths.

### Does ranking in the top 10 get me cited by AI Overviews?

Not reliably. Studies put the share of AI Overview citations that also rank in the organic top 10 anywhere from about 17% to more than half, depending on method. Ranking helps, but it does not decide citation.

### Do I need an llms.txt file?

It is optional. Google's John Mueller has said he knows of no AI service that says it uses it, and no major provider has announced support. It costs little to publish, so add it after access, rendering and content are right.

### How many clicks should I expect from AI answers?

Fewer than from classic results. Pew Research found that people clicked a regular result in 8% of visits with an AI summary against 15% without one, and a link inside the summary in about 1%. Measure visibility and brand mentions as well as clicks.

## Sources and fact check

Checked on 28 Oct 2026.

- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Google Search Central: General structured data guidelines](https://developers.google.com/search/docs/appearance/structured-data/sd-policies)
- [IETF: RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Vercel: The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler)
- [arXiv: GEO, Generative Engine Optimization (Aggarwal et al., KDD 2024)](https://arxiv.org/abs/2311.09735)
- [Pew Research Center: Google users are less likely to click on links when an AI summary appears](https://www.pewresearch.org/short-reads/2025/07/22/google-users-are-less-likely-to-click-on-links-when-an-ai-summary-appears-in-the-results/)
- [Google Search Central Blog: Introducing Search Generative AI performance reports in Search Console](https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports)
- [Search Engine Journal: Query fan-out technique in AI Mode, new details from Google](https://www.searchenginejournal.com/query-fan-out-technique-in-ai-mode-new-details-from-google/)
- [Search Engine Journal: AI recommendations change with nearly every query (SparkToro)](https://www.searchenginejournal.com/ai-recommendations-change-with-nearly-every-query-sparktoro/)
- [Search Engine Journal: Google says llms.txt is comparable to the keywords meta tag](https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/)