How search engines and AI assistants read your website

For twenty years one crawler mattered. Now several do, and some of them answer your customer's question without sending anyone to your site. Here is what that changes and what it does not.

The sitemap

A sitemap is a file listing every page on your site and when it last changed. Without one, crawlers find pages by following links, so anything buried three clicks deep can go unnoticed for months. With one, you have handed over the full list.

The common failure is a sitemap generated once, years ago, still listing pages that no longer exist. That is worse than not having one. A sitemap should be generated from the site's own content so it cannot go stale, which is standard on any modern framework.

robots.txt and crawl budget

A second file tells crawlers what to skip: admin areas, internal search results, duplicate pages, anything that wastes their time.

Crawlers allocate a limited amount of attention per site. Every low value page they spend it on is a page that earns you nothing. Point them at the pages that matter and block the rest.

The page has to contain the content

This is the technical issue behind most invisible websites. If a page arrives as an empty shell and fills itself in with JavaScript, some crawlers and most AI systems see nothing worth indexing.

Server rendering solves it. The server sends finished HTML, so the content is there whether or not any JavaScript runs. That is one of the main reasons I build in Next.js rather than plain React: content pages render on the server by default.

Structured data

Structured data is a block of machine readable information describing the page. That this is a local business, in this postcode, with these opening hours. That this page is an article published on this date. That these are frequently asked questions and their answers.

It is how you earn the extra detail in a search result, and it gives an AI assistant facts to quote instead of an inference. Keep it accurate and matching what is visible on the page. Marking up content that is not there is a penalty risk, not a shortcut.

What AI assistants change

When someone asks an assistant for a plumber in Islington, or which tool suits a ten person team, the answer is assembled from pages it has read. There is no ad slot to buy.

What gets you into that answer is a page that states plainly what you do, who for, and where, in text a machine can read without running code. That is the same work that made you rank on Google. The practice did not change. The audience got larger.

Shortcuts to avoid

Buying links. It works until it does not, and recovering takes longer than the gain lasted.

Hundreds of near identical pages with the town name swapped. These have been filtered out for years. If a page has nothing specific and true to say about a place, do not publish it.

Keyword stuffing. Real visitors leave, and that behaviour is itself a signal.

Anyone guaranteeing a first position. Rankings depend on competitors and on systems nobody outside those companies controls. A guarantee is a sales line.

What actually works

Pages that answer a real question. Titles that describe the page. Fast loading. Internal links that make sense. Accurate structured data. A Google Business Profile that matches the site. It is unglamorous, it compounds, and it does not collapse when the rules change, because none of it was a trick.