Your website ranks on the first page of Google. Then a prospect asks ChatGPT, Perplexity or Google’s AI about exactly what you do, and your organisation is nowhere in the response. You hold the ranking. The citation went to someone else.
This is the gap most teams miss when they think about being found in AI search. The instinct is to write better content, publish more often, and sharpen the messaging. All of that matters. None of it helps if an AI system cannot reach your pages, read them, or make sense of them in the first place.
AI discovery is decided in the build. The structure of your site, the way it is rendered, and the way it is served to crawlers determine whether your content ever enters the pool an AI answer draws from. Good content sitting behind a technical wall is invisible content. This is the layer to take to your web development agency, and the questions below are how to start that conversation.
What makes a site ready for AI search?
A site is ready for AI search when an AI system can do three things: fetch your pages, read the content in them, and understand what that content means. Miss any one, and your pages cannot be cited, however good the writing.
For most of the last two decades, technical SEO meant making a site legible to one crawler, Googlebot, and competing for a ranked link. That link is no longer the finish line. AI search returns a synthesised answer with a short list of citations, so a page can hold its rank and still lose both the click and the mention.
The mechanics that decide citation are familiar. Clear structure, direct answers, and demonstrable expertise are what earn a page a place in an AI answer, the same fundamentals a good website has always needed. What has changed is how many machines read your site and how little most of them tolerate.
The stakes are higher than they look. Even when an AI answer cites a brand’s own website, it’s a small share of citations. McKinsey’s analysis found only around 5 to 10 per cent of the sources AI search references come from a brand’s own website (McKinsey), which means the visibility you control most directly is scarce. A technical fault that removes your pages from that pool is an expensive thing to leave unchecked.
Can AI crawlers read a JavaScript-heavy site?
Mostly, no. Most AI search crawlers fetch your raw HTML and do not run JavaScript. If your main content only appears after the browser executes JavaScript, those crawlers see an empty shell, even when Google renders the same page and ranks it.
Google spent more than a decade building a rendering system that executes JavaScript before indexing a page. The AI crawlers built by OpenAI, Anthropic and Perplexity were built for speed and scale, and most of them do not render at all. They request the page, read whatever HTML the server returns, and move on. OpenAI’s own crawler documentation describes its bots as fetchers of this kind, and independent analyses of large crawl samples have reported the same behaviour.
The AI crawlers built by OpenAI, Anthropic and Perplexity were built for speed and scale, and most of them do not render at all. They request the page, read whatever HTML the server returns, and move on
This matters most for sites built as single-page applications, where a framework such as React, Vue or Angular assembles the content in the browser after the initial load. To a person, the page looks complete. To a crawler that doesn’t run scripts, the initial HTML can be nearly empty, so the content never enters the corpus an AI answer draws from. The page can rank perfectly well on Google, whose renderer fills in the gaps, while remaining blank to every AI crawler that doesn’t.
The fix sits in how the site is built. Serving the main content server-side, through server-side rendering or static generation, or through prerendering for crawlers, puts the content in the raw HTML where every crawler can read it. Modern frameworks support this directly.
There is a quick way to check. Ask your web partner to fetch a key page as a non-rendering crawler, or to view the page source rather than the inspected view, and confirm the main content and headings are present in that raw response. If they are missing, the AI crawlers are missing them too.
How do you control which AI crawlers reach your site?
Through robots.txt, a plain text file at the root of your domain that tells crawlers what they may and may not access. The model is more granular than allow or block. Each AI provider runs several bots with different jobs, and you can permit each one independently. The bot you most want to allow is the one that fetches pages to build AI search answers, because that is what earns citations.
OpenAI is a clear example. Its documentation sets out three crawlers, controlled independently in robots.txt:
| Crawler type | Example (OpenAI) | What it does | If you block it |
| Training | GPTBot | Collects content that may train models | Content is not used for training; search citations unaffected |
| Search and retrieval | OAI-SearchBot | Indexes pages to surface in AI search answers | You lose eligibility for ChatGPT search citations |
| Live user fetch | ChatGPT-User | Fetches a page when a user asks for it | User-initiated visits may be blocked |
Allowing the search and retrieval bot is what makes you eligible to be cited, while the training bot is a separate decision you can make on its own merits. Anthropic, Google and Perplexity run their own equivalents, so the same logic applies across providers.
Setting your robots.txt rules on WordPress
The file lives at yourdomain.com/robots.txt and is the first thing most well-behaved crawlers request. It is organised into groups. Each group names a user-agent, then lists what that agent may and may not reach. By default, anything not disallowed is allowed, so a crawler that is not blocked can already reach you. Naming the AI bots explicitly makes your intent clear and lets you treat training and retrieval crawlers differently:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
WordPress, which is our core platform, handles this file in a way that catches people out. By default, it serves a virtual robots.txt, generated on the fly, with no physical file on disk. A standard install outputs something close to this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: <https://yoursite.com/wp-sitemap.xml>
You can change it in two main ways.
- The simpler one for most teams is an SEO plugin. Yoast SEO, Rank Math and All in One SEO each include a robots.txt editor in the dashboard, so you can add AI crawler rules without touching files over FTP.
- The other is to upload a physical robots.txt to the site root, which overrides the virtual one. Use one method or the other, since a physical file and a plugin-managed virtual file will conflict.
However, one setting can block everything. Under Settings, then Reading, WordPress has a box labelled “discourage search engines from indexing this site.” When it is ticked, WordPress outputs Disallow: / for every user-agent and blocks the whole site. It belongs on staging environments, and it is a common reason a freshly launched site vanishes from search and AI answers. Confirm it is unticked in production. While you’re there, avoid blocking your CSS and JavaScript folders, since crawlers need those files to render the page.
How to check what your site allows
You can verify most of this yourself in a few minutes.
- Read your own file first. Visit yourdomain.com/robots.txt in a browser. It shows exactly what every crawler is being told, and it is the fastest way to spot a stray Disallow: / left over from staging.
- Test specific bots and URLs. Google Search Console has a robots.txt report that shows the file Google last read and any warnings, plus a URL Inspection tool that confirms whether a given page is crawlable. Google retired the old standalone robots.txt tester, so URL Inspection is the current path for per-URL checks. A neutral validator such as TechnicalSEO.com’s robots.txt tool lets you test a URL against a named user-agent such as GPTBot.
- See what a non-rendering bot receives. If your team is comfortable at the command line, fetching a page as a specific crawler shows access and rendering in one step. Running curl -A “GPTBot” <https://yourdomain.com/page> returns the HTML that bot would get. If the main content is missing from that response, the cause is rendering, and the earlier section covers the fix. For a fuller pass, a crawler such as Screaming Frog can walk the whole site under a custom user-agent.
These processes have limits, though. You can confirm whether AI crawlers are allowed and whether your pages are reachable and readable. Confirming whether you are actually being cited in AI answers is harder, since no official tool exists and the products that claim to measure it rely on sample prompts rather than real query data. Treat access and rendering as the things you can verify and fix directly.
You can confirm whether AI crawlers are allowed and whether your pages are reachable and readable. Confirming whether you are actually being cited in AI answers is harder, since no official tool exists and the products that claim to measure it rely on sample prompts rather than real query data. Treat access and rendering as the things you can verify and fix directly.
Two caveats sit underneath all of this. First, an open robots.txt does not guarantee access. A content delivery network, a host-level firewall, or a WordPress security plugin can turn a crawler away even when robots.txt welcomes it, and blocking the sitemap path stops the crawler from discovering your URLs. The robots.txt file and the hosting configuration need to agree.
Second, compliance with robots.txt is voluntary. The major commercial crawlers commit to honouring it and publish verified IP ranges so you can confirm a bot is genuine. For anything that must stay private, use server-side access control, since a crawler directive cannot enforce access on its own.
Do site structure and page speed affect AI visibility?
Yes. A crawler that can reach your pages still has to make sense of them. Clean semantic structure, a logical heading order, healthy page speed, and coherent internal links all make a site easier to crawl, extract and understand, and they feed the traditional ranking that AI answers still draw on.
The factors that help are the ones good front-end teams already care about:
- Semantic HTML and a logical heading order, so a machine can tell a heading from a caption and lift a section as a self-contained answer. The web’s standards bodies have documented this markup for years, through the W3C.
- Place important content early in the page source so it isn’t buried beneath menus and scripts a crawler has to wade through.
- Healthy page speed and Core Web Vitals, which let crawlers work through a site efficiently and give readers a responsive page. Treat this as an enabling condition rather than a proven citation lever, since the honest evidence supports no more than that.
- Internal links that connect related pages, so a crawler can travel through the site and so authority reaches the pages you want cited.
None of this is new. It is the discipline that makes a site fast, accessible and easy to use, applied so that machines can read it too. These are upstream decisions, made in discovery and information architecture before the content exists, and are a subject in their own right.
We saw this play out on the website we designed and built for the Murdoch Children’s Research Institute, one of Australia’s largest medical research institutes. The old site made information hard to find, with poor navigation, weak search and dead links across a large and growing body of research content. We ran the discovery and research, rebuilt the information architecture, and set the technical roadmap and platform to grow with the institute. The result was a structured, contextualised content estate where a visitor can find the right information by browsing or searching. The same structure that makes a large site navigable for a person is what makes it legible to a crawler.
For a purpose-led organisation, that legibility has a direct consequence. When someone asks an AI assistant how to apply for a service, whether they are eligible, or what a policy means, the answer is assembled from sources a machine could reach and parse. If your pages aren’t among them, people who need you are pointed elsewhere.
Does schema markup get you cited by AI?
Schema markup helps machines understand what your page is about, and it does not buy a citation. It clarifies content that is already visible and already good. It cannot make thin content citable, and Google has stated that no special markup is required to appear in its AI features.
Structured data, usually written as JSON-LD, is a way to state plainly what a page contains: that this is an organisation, this is an article, this is the author, this is a question, and this is its answer. It gives a machine an explicit summary rather than leaving it to infer meaning from the prose. The vocabulary is maintained at Schema.org.
Schema clarifies content that already exists and is already clear. It cannot manufacture meaning where the page is thin, and Google has been explicit that no special markup is required to appear in AI Overviews or AI Mode (AI features and your website). The FAQ rich result that many sites added for exactly this reason was restricted to government and health sites in 2023 and removed from general search results in May 2026, though valid FAQPage markup can still help a machine identify a question and its answer (Google’s FAQ documentation). Structured data helps with clarity. It doesn’t enable citations on its own.
One requirement ties directly back to rendering. Schema injected by JavaScript after the page loads is invisible to a crawler that does not run scripts, the same problem as the content itself. For structured data to do its job, it has to sit in the server-rendered HTML. If your answer blocks and page structure are built to be lifted cleanly, schema reinforces them. On its own, it does little.
What to ask your web development agency about AI search
You do not need to build any of this yourself. You need to know which questions to ask, and what a good answer sounds like. These six take the technical layers above and turn them into a short conversation with the people who build and maintain your site, whether that is an in-house web developer or your web development partner.
- Is our main content server-rendered and present in the raw HTML? You want a clear yes, confirmed by fetching a page as a non-rendering crawler or checking the page source rather than the inspected view.
- Which AI crawlers can currently reach us, and is anything blocking them? You want a read on both robots.txt and the hosting layer, with the search and retrieval bots allowed and no firewall rule quietly turning them away.
- Is our structured data in the server-rendered HTML, and does it match what is on the page? You want yes to both. Markup that loads late or contradicts the visible content does not help.
- Are our Core Web Vitals healthy on mobile? You want current field data from real visits, not a single lab score from a good day.
- Is our information architecture and internal linking clean enough for a crawler to move through the site? You want a discoverable sitemap and a logical link structure, so no important page is stranded.
- Is our content organised so a machine can lift a complete answer from one section? You want answer-first sections under clear, specific headings.
None of these requires you to read code. They require your web development agency to show you the site behaves the way an AI crawler needs it to. If the answers are vague, that is useful information in itself.
The last question is the biggest, and it mostly comes down to how each page is written and structured. We cover that in our blog on developing content for AI.
Frequently asked questions about AEO
Will my site show up in AI search if it already ranks on Google?
Not necessarily. Ranking on Google and being cited in an AI answer are now separate outcomes. A page can hold its position and still be missing from AI results, often because AI crawlers cannot render or reach content that Google can. Strong rankings help, though they are no guarantee.
Do AI crawlers read JavaScript?
Most do not. The major AI search crawlers fetch your raw HTML and move on without running JavaScript, so content that only appears after client-side rendering is invisible to them. Google is the exception, since Googlebot renders JavaScript. Server-side rendering or static generation puts the content where every crawler can read it.
What is the difference between GPTBot and OAI-SearchBot?
They do different jobs, as OpenAI’s crawler documentation sets out. GPTBot collects content that may be used to train OpenAI’s models, while OAI-SearchBot indexes pages so they can appear in ChatGPT’s search answers. You can allow one and block the other in robots.txt. Allowing OAI-SearchBot is what keeps you eligible for those citations.
Does schema markup help with AI search?
It helps a machine understand your content, and it does not guarantee a citation. Schema clarifies what is already on the page and helps disambiguate entities. Google has stated no special markup is required for its AI features, and the markup must match your visible content to be useful.
How do I check whether AI crawlers can see my site?
Start by viewing a key page’s source, or ask your developer to fetch it as a non-rendering crawler, and confirm the main content sits in the raw HTML. Then check robots.txt and your firewall or CDN settings to confirm the AI search bots are allowed through.
Is an llms.txt file worth adding?
Not as a priority. As of 2026, no major AI platform has confirmed that llms.txt affects how content is retrieved or cited, and it cannot fix a page a crawler cannot render or reach. Address rendering and crawler access first, and treat llms.txt as an optional signal.
The build is where AI visibility is won
You win AI visibility in the build. The structure, rendering, crawler access, and standards that make a site fast and accessible are the same things that make it legible to the machines now answering your buyers’ questions. Content is necessary. A site that can be reached, read and understood is what lets that content count.
The checklist above is designed to be handed up. If you own the marketing outcome but not the codebase, take those six questions to whoever builds your site and see how clear the answers come back.
Book a call with Darryl Dillon-Shallard, our Chief Growth Officer, to talk through web development for AI search visibility. We build the layer these questions cover as an ISO 27001-certified, WCAG 2.2 specialist team, and we will tell you where your site already stands up and where it doesn’t.
