--- title: "Your agent built a single-page app and the AI crawlers see an empty div" description: "Ask an agent for a website and you will usually get a client-rendered one. The AI crawlers largely do not execute JavaScript, so what they store is an empty div — and one curl command will tell you whether that is your page." url: https://harpywatch.com/blog/an-empty-div-to-the-crawler kind: article author: "The HarpyWatch team" published: 2026-09-02T18:06:09.048778+00:00 tags: ["AI Agents", "Monitoring", "SEO"] publisher: "HarpyWatch" --- # Your agent built a single-page app and the AI crawlers see an empty div Run this against your own marketing site: ``` curl -s https://example.com/ | head -c 2000 ``` You are looking for one thing: is your actual copy in there? The headline, the paragraphs, the product names. If what comes back is a ``, a couple of script tags, and `
`, then that is what the AI crawlers see when they ask for your page. Not a browser. A client. The distinction matters more every month. ## Why the page came out this way Ask an agent for a landing page and you will very often get React, or Next, or Vite with a client-side router. That is not mysterious: those are the stacks with the most public code, the most tutorials and the most StackOverflow answers, so they are the stacks a model has seen most. It is a reasonable default too — the patterns are well documented and the result works when a person loads it. The prompt was "build me a landing page". The answer was a component tree that assembles itself in the browser. Nobody in that exchange made a decision about rendering — it arrived as the shape of the answer — and the review that followed happened in a browser, where the difference does not show. ## What actually reads HTML now Here is the mechanism, stated as precisely as we can without overclaiming. **Googlebot renders JavaScript.** It has done for years. Pages go into a rendering queue, get executed in a headless Chromium, and the rendered content is indexed. This is real and it works. Two caveats survive: rendering happens on a delay after the initial fetch, and it is subject to a budget — a large site that is entirely client-rendered is asking for a great deal more crawl resource than one that ships HTML. For a small site the delay is usually the only cost. **The AI crawlers largely do not** — and this is an observation about what they fetch, not a claim about how they are built, because nobody outside those teams knows that. GPTBot, ClaudeBot, PerplexityBot and CCBot (Common Crawl, a major input to open training corpora and to several retrieval pipelines) request a URL and generally do not follow it with the pattern of subresource requests a browser makes. If the server sends a shell that needs JavaScript to become a page, the shell is what gets stored. This is not a fixed law and we would not want to state it as one. Some of these operators run a rendering step for some requests, retrieval products often fetch a page live at query time with different machinery from their training crawler, and none of it is announced when it changes. Which is the argument: **do not assume execution, and do not assume it stays the same. Measure what your server sends.** The consequence is not a ranking penalty. It is simpler and worse. Your page exists in these systems as a document with no content in it. When someone asks an assistant what your product does, whatever it says came from somewhere else — a directory listing, a competitor's comparison page, a forum thread. ## The same failure, three other places Client-only rendering breaks more than crawlers, and the other cases are useful because they are easier to test. - **Link previews.** Slack, WhatsApp, iMessage and LinkedIn unfurl links by fetching HTML and reading meta tags. If the title and description are set by JavaScript after mount, the preview is blank or shows the framework default, so every link anyone shares to your site looks broken. - **Fetch-and-parse clients generally.** Feed readers, archive crawlers and many security scanners fetch and parse without executing. Some archive and scanning tools do drive a browser; most of the long tail does not. - **The first paint for a real person on a bad connection.** A shell plus a bundle plus a data fetch is three round trips before there are words on the screen. This is a user experience problem before it is anything else. ## What to check, and how The useful question is not "does my site work" — it does — but "do these two things agree": 1. What the server sends on a plain HTTP request. 2. What exists after a browser has run the JavaScript. If they differ substantially, the gap is exactly the content that does not exist for a non-executing client. You can do this by hand with curl, and you should, once, right now, for your most important three pages. `curl -s | grep -i ""` is a five-second answer to a question most teams have never asked. Doing it continuously is what monitoring is for. We build one, so the two checks below are ours — the point is the pairing, which you could assemble from curl and a headless browser yourself: **`dom-content`** fetches the URL as an HTTP client and asserts on what came back — a string is present, a selector matches, a phrase you expect is in the document. No JavaScript is executed. That is the point: it is a measurement of what a client that does not execute JavaScript receives. **`browser-render`** loads the page in a real browser and asserts against the result after scripts have run. Running both against the same URL turns an invisible architectural decision into a visible one. `browser-render` green and `dom-content` red is not two failures. It is a single sentence: *this page only exists if you execute JavaScript.* Sometimes that is fine — an authenticated dashboard has no crawler audience. For a pricing page it is a problem you would want to have chosen deliberately. ## What we did about our own This blog is server-rendered. The rest of HarpyWatch's front end is an Angular application, because it is an authenticated grid of check results that no crawler will ever see. Same codebase, two rendering strategies, because the two surfaces have different audiences. That is not a clever trick; it is the ordinary answer. It is worth mentioning only because the question has to be asked per surface, and the failure happens when one answer gets applied to everything by default. ## The code change is smaller than the decision Most frameworks that produce this problem also solve it. Next has server components and static generation; Nuxt, SvelteKit, Remix and Astro all ship HTML without needing a client to assemble it. Angular has SSR. Nobody has to leave their stack. That is not the same as free. Turning on server rendering in an application that was not written for it brings hydration mismatches, code that assumed `window` existed, a Node process to run and cache, and a new set of things that can be stale. For a page whose job is to display text, this is usually contained. For an application with third-party scripts and an authenticated shell, it is a project, and the scripts are where it hurts. What has to change is that the decision gets made. An agent asked to build a page will not ask you whether search engines and language models should be able to read it, because that is not a coding question — it is a question about who the page is for. That one is still yours. Then check that the answer stayed true. A rendering strategy is a thing an agent can quietly change in a refactor six weeks later, and the only symptom is that `curl` gets shorter.