JavaScript SEO: when the engine sees an empty page

You open your site. Everything in place, prices load, filters work, the catalogue scrolls. Fine.

Now do something that takes ten seconds. Right click, view page source, and use Ctrl+F to search for a phrase from your own copy. Not the menu, a phrase from the substance.

Half of you will not find it.

That does not mean the site is broken. It means the text is not in the page the server sends. It exists in your browser’s mind, once your browser has run the code. And whoever does not run code does not see it.

What Google does, in its own words

Let us not guess. Google’s documentation says it plainly: “Google processes JavaScript web apps in three main phases: 1. Crawling 2. Rendering 3. Indexing”.

The second is the critical one. The bot downloads the HTML first. If the HTML is nearly empty, it queues the page for rendering, and later, “once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript”. Headless Chromium is Chrome without a screen, running automatically.

Read that last sentence again, because the whole matter hides in it. Once resources allow. The same page puts it even more bluntly: a page “may stay on this queue for a few seconds, but it can take longer than that”.

So the answer to “does Google see my JavaScript?” is not yes or no. It is yes, later, if it gets round to it. For an article you wrote last year, fine. For a product page whose price changed this morning, less so.

And everybody else

Here the landscape changes, and it changes a great deal.

Vercel, the company behind Next.js, published a measurement from its own network, not an estimate, on 17/12/2024. It tracked crawler traffic for months on nextjs.org and across the Vercel network, and validated the findings on two further sites. The finding, verbatim:

“None of the major AI crawlers currently render JavaScript. This includes: OpenAI (OAI-SearchBot, ChatGPT-User, GPTBot), Anthropic (ClaudeBot), Meta (Meta-ExternalAgent), ByteDance (Bytespider), Perplexity (PerplexityBot).”

And a detail that makes it more interesting: the bots do fetch JavaScript files, in 11.50% of requests for ChatGPT and 23.84% for Claude. They simply do not execute them. They pay for the download and get nothing from it, exactly as you do.

Two caveats, because this is how numbers should be written. First, the measurement carries a date, December 2024. Somebody may have changed this since, and if they do, we will write about it. Second, it is a company measuring its own traffic, not an independent auditor. It remains the most substantial measurement I have found, and it is a great deal better than the hundred undated reproductions of it in circulation.

The practical conclusion is one sentence: Google forgives you, the answer engines (AI) do not. And traffic from the latter is growing, as we have written elsewhere.

The two-minute check

Three steps, in this order.

First, view source. Right click, view page source, Ctrl+F, a phrase from your copy. That is what the server sends. Careful: “inspect element” will not do for this check, because it shows you the page after your browser has built it. It is the most common misunderstanding and it leads to the wrong conclusion.

Second, switch JavaScript off. In Chrome, developer tools, Ctrl+Shift+P, type “disable JavaScript”, reload. Whatever is left on screen is roughly what an engine that does not render sees. The surprise here is usually considerable.

Third, ask Google itself. In Search Console, the URL Inspection tool, or the Rich Results Test. The documentation says that there you see “loaded resources, JavaScript console output and exceptions, rendered DOM”, meaning the page as Google finally built it. If your text is missing there too, this is not a queueing matter, it is an error.

The four that break most often

Links that are not links. Google writes it without room for misreading: it “can only discover your links if they are a HTML elements with an href attribute”. An element that merely reacts to a click (a div with an onclick), with no real address behind it, is a button, not a link. It works perfectly for the visitor and does not exist for any engine. It is the number one finding in every audit I run.

Content behind a click. Tabs, accordions and “show more” that fetch the text at the moment the user clicks. If the text arrives with a request after the click, it does not exist for anyone who does not click. And bots do not click. The fix is not to open everything up; it is for the text to be in the HTML and simply not visible.

Infinite scroll. The catalogue of 600 products that loads twenty at a time as you scroll down. The bot does not scroll. It sees the first twenty and leaves. You also need pagination with real URLs underneath, even if no user ever touches it.

The fake 404. When /product-that-does-not-exist answers with status 200 (the server’s “all good”) and shows “not found” through JavaScript, the engine receives a 200 from the server while the page is an error page. Google usually works this out and records a soft 404, wasting crawl budget on addresses that do not exist in the meantime. Engines that do not render JavaScript do not even have that ability. Google suggests either a “JavaScript redirect to a URL for which the server responds with a 404 HTTP status code”, or adding a noindex tag through JavaScript.

The fix, which is not to rewrite anything

One thing: have the server send finished HTML.

In modern frameworks this is called server side rendering or static generation, and most of the time it is a setting rather than a rewrite. For older applications there is prerendering, meaning a layer that prepares the page once and serves the result. The purpose is always the same: by the time the request arrives, the text is already there.

The gain is not only with the engines. It is in speed too, because the visitor sees text before the JavaScript has downloaded and run, and that is measured in Core Web Vitals. Two problems, one job.

And so as not to be a hypocrite: this site is static. Every page exists in full before anybody asks for it. We did not do it for the answer engines; we did it because it is the simplest thing that works, and it happened to also be the right thing.

When none of this matters

I do not want you changing things for no reason.

Whatever sits behind a login is crawled by nobody. Dashboards, internal tools, the customer area, the basket, the checkout. There JavaScript does exactly the job it was built for and there is no issue at all.

The question applies only to what you want found: home page, services, products, categories, articles, store pages. If those exist in the HTML, let the rest be built however it likes.

The one minute worth spending

Do the first check now, while you are reading this. View source, Ctrl+F, a phrase from your home page.

If you find it, you are fine and you can close the article. If you do not, nothing is on fire today, but you now know why some of your pages are slow to be indexed and why no answer engine mentions you when people ask about your line of work. The same argument explains what happens outside Google.

If the phrase was not there, or some of your pages are slow to be indexed, this is what we check: we look at the pages you want to be found and at how much of them actually reaches the engines. You get a list of what we found and what we recommend, in order of priority. Send me the site address and two or three pages you care about through the contact form. This check is part of our SEO and GEO service.

The whole picture: this article covers one part of a larger subject. If you are after the whole of it, from beginning to end, it is in the guide to answer engines.