The Bots Have Changed and Your Site Has Not
For twenty years, technical SEO meant preparing your site for one very sophisticated reader: Googlebot. It renders JavaScript, waits for your hydration, and forgives a lot.
The new readers forgive nothing. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and a dozen smaller AI crawlers are hitting production sites every day, and most of them share two traits: they do not execute JavaScript, and they spend seconds, not minutes, deciding whether your site is worth ingesting.
I have now done this preparation work on several production sites, including a full association site rebuild where we went from an empty client rendered shell to fully prerendered pages. This is the checklist I actually use.
Step One: See What the Machines See
Fetch your homepage the way a bot does. In a terminal, curl your URL and read the raw HTML. No browser, no JavaScript.
Ask one question: is the content there? Not the meta tags, the content. On one rebuild I audited, every route had perfect titles, descriptions, and JSON-LD attached to a completely blank body, because the site was a single page app that shipped an empty root div. Human visitors saw a beautiful site. Machines saw a well labeled void.
If that is your situation, nothing else on this list matters until it is fixed.
Step Two: Serve Real HTML
You have three paths, in descending order of preference:
- Static generation or server side rendering. Frameworks like Next.js and Astro emit real HTML by default. If you are choosing a stack today, choose one of these.
- Prerendering on top of an SPA. If a rebuild is off the table, add a server layer that renders each route's actual content into the initial HTML response. On the association project we wrote a prerender module that mirrors the React pages server side; the client app mounts over it, browsers notice nothing, and bots get everything.
- A prerendering service. Third party middleware that detects bots and serves them a rendered snapshot. Workable, but you now own a second cache and a user agent detection problem.
- Organization and Person schema with sameAs links to your real profiles, connecting your content to a verifiable entity.
- Article schema with real authors, dates, and images, on every post.
- Event, FAQPage, and JobPosting schema where the content genuinely matches. Machines cross check schema against visible copy, and mismatches destroy trust.
- Fast responses. Static and edge cached pages get crawled deeper than slow dynamic ones.
- Clean status codes. A wrong URL must return a real 404. On one audit we found every junk URL answering 200 with an empty page, which invites a crawler to index garbage and dilute the real content.
- No orphan pages. If your internal links only exist in a JavaScript menu, bots see none of them. Ensure the served HTML contains real anchor tags to your important pages.
- A truthful sitemap. Every URL in it should return 200 with unique content, and nothing important should be missing.
- Most AI crawlers do not run JavaScript. Curl your pages and confirm the content exists in raw HTML.
- Prerender or server render everything important. Meta tags on a blank body fool no one.
- Make an explicit robots.txt decision about AI crawlers, and repeat your Disallow rules inside every named crawler group.
- Add llms.txt, keep schema truthful, serve fast pages with honest status codes.
- The payoff is citations, assistant referrals, and accurate AI answers about your brand.
While you are in there, check word count. A page that renders 40 words of navigation around a JavaScript dependent body is still functionally invisible.
Step Three: Write an Explicit Robots Policy
Every major AI crawler respects robots.txt. That gives you an actual policy decision to make: do you want to be in training data and answer indexes, or not?
For most brands the answer is yes, visibility is the point. Then say so explicitly. List the crawlers you welcome by name, and remember a subtle trap we hit in production: if you give a named crawler its own Allow group, it obeys only that group and inherits none of your general Disallow rules, so you must repeat the sensitive path exclusions inside every named group. A crawler follows exactly one group, and a permissive group silently voids the protections you wrote for everyone else.
Step Four: Add llms.txt
The llms.txt convention is simple: a plain text file at the root of your domain that gives language models a curated map of your site: what it is, who it belongs to, and where the canonical pages live. It costs an hour to implement. On sites with a database behind them, generate it from the same content source as your sitemap so it never drifts.
Adoption is early, which is exactly why it is worth doing now. Early infrastructure adopters get outsized attention from systems that are actively looking for structured guidance.
Step Five: Make Your Schema Load Bearing
Structured data used to be about rich snippets. Now it is how entity based systems understand who is speaking:
I go deeper on the entity side in my personal SEO guide, which is the same discipline applied to a person instead of a company.
Step Six: Respect the Crawl Budget
AI crawlers are impatient and their operators pay per token. Help them spend efficiently:
What This Buys You
This is not speculative optimization. Sites that did this work are already collecting the returns: assistant referral traffic, citations in AI answers, and accurate representation when someone asks a model about the brand. I laid out the strategy layer in my answer engine optimization guide; this piece is the engineering underneath it.
Key Takeaways
The last platform shift punished sites that ignored mobile. This one will punish sites that machines cannot read.