Anthony McGrathHow to Prepare Your Website for AI Search
Before redesigning your website, audit how conversational engines describe your business. Here is how to fix outdated data and check crawler access.
Quick answer
AI search pulls answers from existing web pages, meaning outdated company details or blocked crawlers can wipe out your visibility. To fix this before a redesign, audit your current brand queries across platforms, check your robots.txt permissions for search bots, and publish accurate, unique product data that answer engines can read.
Key takeaways
- Run test queries across conversational engines to detect outdated product details.
- Ensure your robots.txt file allows search crawlers to access core pages.
- Use Google Search Console reports to track generative search visibility.
A guide published on note on 20 October 2026 outlined how businesses should audit their presence in AI search before launching a website redesign. Many business owners discover that chatbot engines either invent details about their products or quote retired pricing.
If you are planning a rebuild, fixing these gaps early saves painful redesign rework later.
How do you audit your site for answer engines?
Start by querying tools like ChatGPT, Claude, and Perplexity with simple brand questions. Ask what your business sells and how to choose products in your sector. Record the responses and note where the engine retrieved outdated details.
Author Masaru Kondo noted four main reasons a brand gets missed:
- Crawlers are blocked from reading the site.
- Published information is incomplete or outdated.
- The content lacks original insight.
- Old details on third-party directories confuse the models.
Check your robots.txt file to ensure search bots can access your core pages. OpenAI and Anthropic use separate crawlers for training models and live web fetching. Blocking training crawlers will not hurt your search visibility as long as you allow search bots through.
Google stated in 2026 that special files like llms.txt are unnecessary. Standard optimisation remains the baseline. If you want to understand how these strategies fit together, read my guide on the difference between SEO, GEO and AEO.
To monitor progress, Google Search Console rolled out generative impression reporting to all websites by late August 2026. This allows you to track whether your pages appear in conversational overviews without paying for third-party tools. Fast technical foundations also matter for crawler efficiency, which you can test with our free site speed checker.
What it means for you
- Run three brand queries across two conversational search engines this week to spot incorrect product facts.
- Review your robots.txt file with your developer to ensure live search user-agents are not accidentally blocked.
Questions and answers
Do I need an llms.txt file for search engines?
No. Google confirmed in 2026 that files like llms.txt are unnecessary for search retrieval. Standard SEO best practices, structured data, and clear page content provide all the context search crawlers require.
Can I block AI training without losing search visibility?
Yes. Major AI providers such as OpenAI and Anthropic separate their web crawlers into distinct user-agents for training models and live search indexing. You can block training bots in robots.txt while allowing search bots.
How can I track generative search traffic?
You can track impressions from conversational overviews directly in Google Search Console, which completed its rollout of generative feature reporting to all verified sites in late August 2026.
