The value of bot experience for AI search

As LLMs and AI chatbots bring us into a new paradigm of Search, a core pillar of technical SEO remains critical: site health.

The concept of site health reflects the experiences bots have on your site. For example, how easy it is for Googlebot to crawl, render, index and rank content on your site. Most health factors also matter to users, like slow loading speed, broken images or broken links. But some of them, especially crawling and rendering, are exclusive to bots.

Bot Experience (BX)

In the conversation around user experience, we often ignore bot experience. But bots shouldn’t be second-class citizens on our site.

Any issue related to crawling, rendering and indexing impacts the Bot Experience:

• Images

• Links

• Canonicalization

• Status Codes

• Rendering approach

• Critical Rendering Path

• Sitemaps

• Core Web Vitals

• Duplicate Content

• JavaScript

One way to measure Bot Experience is Ryte’s WUX Score, a single number from 1 to 100 that reflects technical site health. The score comes together by a list of weighted issues, for example: 

• Broken pages, images and files

• Links to 404 pages

• Heavy pages

• Duplicate content

• Broken redirects

The weight of an issue is calculated by the number of times it occurs and its impact on organic traffic. The WUX score formula is 100 minus the impact of all issues. Optimizing the score means optimizing user and bot experience.

While I’m generally against a checklist approach to SEO, they can simplify prioritization and tracking in tech SEO specifically. Where SEO checklists miss the (often) unique landscape of target queries, SERP layout, business model and competitors, checklists for UX and BX are valuable because we can all agree that certain issues, like links to 404 pages, are bad.

Besides being able to understand historic trends, the visual nature makes it easier to get executive buy-in for tech SEO projects.

3 reasons why you should care

1/ All too often, I get questions about the value of fixing tech SEO issues:

“You want us to fix one thousand 404 pages - how much revenue is this going to make us?” 

And, all too often, the answer is… crickets.

It doesn’t have to be this way.

Not all processes at companies make money. Some keep the company running or minimize the downside in case of emergency. An accountant would call Tech SEO fixes operational expenses or OpEx. It’s the cost you pay to run your business. In SEO, it’s the same idea: having a robust technical base allows you to do fun things with content, links and user experience.

The key to running with that concept is selling it well. The way to do it is highlighting that SEO traffic needs to be defended and the cost of it. Other than common believe, you won’t keep the same level of SEO traffic if all SEOs went on vacation for a few months. The cost of keeping the traffic you have is the opportunity of Bot Experience optimization.

2/ Analyses of Search Generative Experience (SGE) have shown that technical issues (example: Onely), like unrenderable JavaScript, reduce your chances of showing up in SGE. It makes sense when you think about it, but even issues that don’t prevent Googlebot from indexing content seem to have a negative impact.

Even after years of optimizing their crawler and helping SEOs build better sites, Google still has issues with Javascript. Despite all the hype around AI and the future of search, it doesn’t look like we’ll get to a place where we can avoid technical optimization.

3/ Technical issues like broken links, 404 pages or broken images might not be a complete roadblock for Google, but they surely provide a poor user experience.

As we’ve learned from documents revealed in Google’s current antitrust case, positive user behavior (clicks, hovers, bounces, pogo-sticking, etc.) has been Google’s main element of ranking for decades. Only recently have they discovered better ways to understand content and backlinks or mentions.

So, having lots of technical issues on your site can deteriorate the user experience, like slow page elements, and lower ranks as a result - even when Google can work around them.

Learn more about Ryte’s WUX Score HERE.