Your agent built a single-page app and the AI crawlers see an empty div
Ask an agent for a website and you will usually get a client-rendered one. The AI crawlers largely do not execute JavaScript, so what they store is an empty div — and one curl command will tell you whether that is your page.
Run this against your own marketing site:
curl -s https://example.com/ | head -c 2000
You are looking for one thing: is your actual copy in there? The headline, the
paragraphs, the product names. If what comes back is a <head>, a couple of
script tags, and <div id="root"></div>, then that is what the AI crawlers see
when they ask for your page.
Not a browser. A client. The distinction matters more every month.
Why the page came out this way
Ask an agent for a landing page and you will very often get React, or Next, or Vite with a client-side router. That is not mysterious: those are the stacks with the most public code, the most tutorials and the most StackOverflow answers, so they are the stacks a model has seen most. It is a reasonable default too — the patterns are well documented and the result works when a person loads it.
The prompt was "build me a landing page". The answer was a component tree that assembles itself in the browser. Nobody in that exchange made a decision about rendering — it arrived as the shape of the answer — and the review that followed happened in a browser, where the difference does not show.
What actually reads HTML now
Here is the mechanism, stated as precisely as we can without overclaiming.
Googlebot renders JavaScript. It has done for years. Pages go into a rendering queue, get executed in a headless Chromium, and the rendered content is indexed. This is real and it works. Two caveats survive: rendering happens on a delay after the initial fetch, and it is subject to a budget — a large site that is entirely client-rendered is asking for a great deal more crawl resource than one that ships HTML. For a small site the delay is usually the only cost.
The AI crawlers largely do not — and this is an observation about what they fetch, not a claim about how they are built, because nobody outside those teams knows that. GPTBot, ClaudeBot, PerplexityBot and CCBot (Common Crawl, a major input to open training corpora and to several retrieval pipelines) request a URL and generally do not follow it with the pattern of subresource requests a browser makes. If the server sends a shell that needs JavaScript to become a page, the shell is what gets stored.
This is not a fixed law and we would not want to state it as one. Some of these operators run a rendering step for some requests, retrieval products often fetch a page live at query time with different machinery from their training crawler, and none of it is announced when it changes. Which is the argument: do not assume execution, and do not assume it stays the same. Measure what your server sends.
The consequence is not a ranking penalty. It is simpler and worse. Your page exists in these systems as a document with no content in it. When someone asks an assistant what your product does, whatever it says came from somewhere else — a directory listing, a competitor's comparison page, a forum thread.
The same failure, three other places
Client-only rendering breaks more than crawlers, and the other cases are useful because they are easier to test.
- Link previews. Slack, WhatsApp, iMessage and LinkedIn unfurl links by fetching HTML and reading meta tags. If the title and description are set by JavaScript after mount, the preview is blank or shows the framework default, so every link anyone shares to your site looks broken.
- Fetch-and-parse clients generally. Feed readers, archive crawlers and many security scanners fetch and parse without executing. Some archive and scanning tools do drive a browser; most of the long tail does not.
- The first paint for a real person on a bad connection. A shell plus a bundle plus a data fetch is three round trips before there are words on the screen. This is a user experience problem before it is anything else.
What to check, and how
The useful question is not "does my site work" — it does — but "do these two things agree":
- What the server sends on a plain HTTP request.
- What exists after a browser has run the JavaScript.
If they differ substantially, the gap is exactly the content that does not exist for a non-executing client.
You can do this by hand with curl, and you should, once, right now, for your
most important three pages. curl -s <url> | grep -i "<your headline>" is a
five-second answer to a question most teams have never asked.
Doing it continuously is what monitoring is for. We build one, so the two checks below are ours — the point is the pairing, which you could assemble from curl and a headless browser yourself:
dom-content fetches the URL as an HTTP client and asserts on what came
back — a string is present, a selector matches, a phrase you expect is in the
document. No JavaScript is executed. That is the point: it is a measurement of
what a client that does not execute JavaScript receives.
browser-render loads the page in a real browser and asserts against the
result after scripts have run.
Running both against the same URL turns an invisible architectural decision into
a visible one. browser-render green and dom-content red is not two failures.
It is a single sentence: this page only exists if you execute JavaScript.
Sometimes that is fine — an authenticated dashboard has no crawler audience.
For a pricing page it is a problem you would want to have chosen deliberately.
What we did about our own
This blog is server-rendered. The rest of HarpyWatch's front end is an Angular application, because it is an authenticated grid of check results that no crawler will ever see. Same codebase, two rendering strategies, because the two surfaces have different audiences.
That is not a clever trick; it is the ordinary answer. It is worth mentioning only because the question has to be asked per surface, and the failure happens when one answer gets applied to everything by default.
The code change is smaller than the decision
Most frameworks that produce this problem also solve it. Next has server components and static generation; Nuxt, SvelteKit, Remix and Astro all ship HTML without needing a client to assemble it. Angular has SSR. Nobody has to leave their stack.
That is not the same as free. Turning on server rendering in an application that
was not written for it brings hydration mismatches, code that assumed window
existed, a Node process to run and cache, and a new set of things that can be
stale. For a page whose job is to display text, this is
usually contained. For an application with third-party scripts and an
authenticated shell, it is a project, and the scripts are where it hurts.
What has to change is that the decision gets made. An agent asked to build a page will not ask you whether search engines and language models should be able to read it, because that is not a coding question — it is a question about who the page is for. That one is still yours.
Then check that the answer stayed true. A rendering strategy is a thing an agent
can quietly change in a refactor six weeks later, and the only symptom is that
curl gets shorter.
Cite this
The HarpyWatch team. “Your agent built a single-page app and the AI crawlers see an empty div”. The Nest, HarpyWatch, 2 September 2026 UTC. https://harpywatch.com/blog/an-empty-div-to-the-crawler
More from The Nest
The agent will make the check pass
Given a red signal, an agent optimises for the signal. That is not a flaw in any particular tool — it is what optimisation means, and it changes which signals are worth having.
Your monitor cannot live in the thing it monitors
A health check inside your own stack goes silent at exactly the moment it had something to say. Here are the failure domains an internal check shares with the site, and what an external one sees instead.
The deploy succeeded. That is not the same as the deploy working
A green pipeline reports that a process started. It is compatible with an empty database, a missing environment variable, a stale CDN and every real route returning 500. Only a request from outside settles it.