What AI crawlers see when your page needs JavaScript

A page that assembles itself in the browser can be perfectly visible to you, to your analytics and to Google, and still reach an answer engine as an empty shell with a loading spinner in it. The frustrating part is that almost nobody will tell you which of those happened. Two vendors document that their crawler renders. The others do not address the question in their crawler documentation at all, which is not the same as a no — it means you have to find out yourself.

What does “rendering” actually mean here?

Fetching and rendering are separate steps. A fetch retrieves the HTML your server sent. Rendering executes the scripts that HTML references, waits for them to modify the document, and reads the result. Google describes its own pipeline as three stages — crawl, render, index — with a headless Chromium doing the middle one, and is explicit that pages can sit in the rendering queue for longer than a few seconds before that happens.

Everything downstream depends on which of those two artefacts an engine kept. If it stored the fetched HTML, your content exists only to the extent that it was in the response your server produced. If a framework injected the article body after hydration, the engine has your navigation, your footer and nothing worth quoting.

Which crawlers are documented as rendering?

Agent	What the vendor documents
Googlebot	Renders in headless Chromium as a distinct queued stage after the crawl
Applebot	Renders page content in a browser; blocked scripts or styles can compromise it
OpenAI, Anthropic, Perplexity, Meta agents	Crawler documentation does not address JavaScript execution

Google’s own JavaScript guidance closes with the sentence that matters most here: server-side or pre-rendering is still a good idea, because not all bots can run JavaScript. That is the search engine with the most capable renderer in the world telling you not to depend on having one.

How do I test an agent whose vendor says nothing?

Your request logs answer this without any cooperation from the vendor, because rendering is not silent. A crawler that executes your page has to go and get the things the page asks for.

1. Isolate one agent’s requests for a single page URL, by user-agent and ideally after verification, so a spoofed header does not pollute the result.
2. Look at what else that same client fetched in the following seconds. A renderer requests your bundle, your stylesheet, your fonts and usually a handful of API endpoints. A plain fetcher requests the HTML document and stops.
3. Compare the two sets across several pages. One quiet page proves nothing; a consistent pattern of HTML-only requests across an archive is a strong signal, and a consistent pattern of bundle requests is a stronger one in the other direction.

There is a cruder check that takes a minute: curl the page with scripting absent, and read what comes back. Whatever is in that response is the floor — the least any crawler can have received. If your lead paragraph is not in it, no amount of vendor documentation will save you.

What does a non-rendering crawler get from a client-side page?

Usually a valid, empty document. The title and meta description are in the head and survive, which is why these pages often look fine in a link preview and in a sitemap audit. The body is a mount point. An engine that chunks that document for retrieval finds nothing self-contained to store, so the page is not so much ranked badly as absent from consideration.

Partial failures are more common and harder to spot than total ones. A server-rendered article with client-rendered comparison tables, pricing, specifications or FAQ accordions loses precisely the structured, quotable material an answer engine most wants. The page reads as complete to you and arrives as prose without evidence.

Does blocking scripts or styles in robots.txt matter?

For the renderers, yes, and Apple says so directly: if the resources needed to render are disallowed, rendering may be compromised, and publishers should make sure everything a user needs to display the page is available to Applebot. A Disallow: /assets/ written years ago to keep bots out of a static directory is a common and entirely invisible cause of this.

For non-rendering agents it makes no difference at all, because they were never going to ask for those files. The asymmetry is worth remembering: the rule costs you something with the two vendors whose crawlers are most capable, and buys you nothing with the rest.

What about content behind an interaction?

Treat anything that appears only after a click as absent. Tabs, accordions, “read more” toggles, infinite scroll and modal dialogs are all cases where the content may be in the DOM or may be fetched on demand, and only the first survives. The reliable version is to have the text present in the response and let the interaction control visibility rather than existence — an accordion whose panels are in the HTML and merely collapsed is fine; one that fetches its panel on open is not.

This is the same constraint as writing passages that stand on their own, approached from the delivery side rather than the editorial side. A fragment cannot be lifted out of a page it never reached.

Does this affect advertisers as well as publishers?

Yes, and it is usually the more expensive failure. A brand that is losing in AI answers often has a landing page that does not exist as text: the product comparison, the specifications and the pricing are all injected client-side, so the page an engine can quote contains a headline and a call to action. Before concluding that you need presence in somebody else’s content, check that your own pages arrive as content. Running the curl test against your three most important URLs takes about five minutes and occasionally ends the investigation on the spot.

What are the limits of this?

Log evidence tells you what an agent did, not what it is capable of, and vendors change their infrastructure without announcing it. An agent that fetched only HTML last quarter may render this quarter. Absence of bundle requests is also ambiguous where a CDN serves your assets from a different hostname whose logs you are not reading.

The advice survives the uncertainty, though, because it is asymmetric. Server-rendering the substance of a page costs you nothing with a crawler that renders and saves you everything with one that does not. There is no corresponding scenario where client-side assembly wins.

Bottom line

Do not try to establish which engines render. Establish that your important text is in the response your server sends, and the question stops mattering. Then check your robots.txt is not blocking the assets the two documented renderers need, and check the crawler reference for which agents are fetching you at all — an engine that never requested the page cannot have rendered it either.