SEO Split Testing vs. Standard A/B Testing: What's the Difference?

While the names are similar, there are a few key differences between standard A/B testing and SEO split testing.

Here’s the quick rundown …

Standard A/B testing has been around in the online (and physical!) world for decades. For web pages, designers will often present multiple design variations of a page for user feedback on usability and aesthetics. 

In recent years, A/B testing has become more sophisticated. Companies like Optimizely or Adobe (Test and Target) use JavaScript to split user traffic into groups to present these variations that can be as simple as button colors and copy changes, to more complicated layout changes. 

This type of testing is also known as user A/B testing, user split testing, conversion rate optimization testing (CRO), or multivariate testing.

SEO split testing, while similar in that page variations are being tested, has some differences. These changes must be tested in a way that search engines can crawl, index, and surface them in search results pages. 

In order to “split” test the changes, the changes must be made to one or more pages and measured against another similar page or group of pages. 

Recommended Reading: The Right Way to Do SEO Split Testing

For SEO split testing, both the test page (that with the change) and the control page (the unchanged page) are fully accessible to all search engines and all users.

This does not measure what the user does when visiting the page — it tests how search engines respond to the page differences, not something like conversion rate.

Here are some SEO testing ideas to consider. 

How Traffic is Split Between the Tests

Standard A/B testing sends a portion of the audience to one version of a page (the control), and the other portion to a second version of that page (the test).  

Users are often given a cookie to ensure they consistently sees either the control version or the test version. In order to prevent confusion or the illusion of cloaking and duplicate content, we typically hide the test variant from search engines since they do not accept cookies.

Recommended Reading: Does A/B Testing Negatively Affect SEO? 

With SEO split testing, pages are split into “groups.”

Users and search engine bots can actually see both versions of the page. You measure the different ways in which the search engine ranks and displays the test pages, and how the user interacts with those changes on the SERPs (i.e. whether or not they click) instead of how they act on the page itself.

Google’s Ability to See the Test/Content

Standard split testing is purposely hidden from Google, while SEO A/B testing is made to be indexable. 

If you don’t hide the user testing from search engines, the test content could be flagged as cloaking or duplicate which could have negative consequences like lower ranking or even a search engine penalty — both of which result in reduced traffic. 

Or, since search bots cannot accept the test cookie that locks them into a test or control group, the content could simply flip-flop back and forth between test and control for each Googlebot visit, creating uncertainty that would also negatively impact ranking.

On the other hand, SEO testing has to allow search engines to crawl both versions, so you can test how they are indexed and ranked, and potentially clicked by search engine users. 

Therefore, users can encounter the test and control versions, since we want the search bots to do so. This can potentially confuse real users so you may need to employ your changes carefully so the user does not become frustrated with the different experiences for the same type of page.

How Results are Measured

Standard A/B testing measures how a person reacts to a change when landing on the page. The test subject is the user. 

Did they interact more favorably with the new test version versus the users who saw the control version? This could be clicking on a blue versus red CTA button, for example. 

An SEO split test sets out to measure how both search engines react to the page and how users react to the result when found in a SERP.

The first test subject in this case is a search engine. How did they respond to the test pages? Did they rank them higher or lower? Did Google rewrite the title or snippet? Did Google render our rich snippets, if that was our goal?

The second test subject is the user and is most often measured in traffic. Did they click through more for the test page versus the control pages? 

Sometimes those differences include new language, CTAs in the metadata, or even rich results. Search engines will often react in multiple ways to changes. 

For example, they may both increase the rank as well as display a new CTA. If traffic increased, it may be difficult to determine if it was a result of increased rank or the CTA. Therefore, it may require more nuanced SEO split testing to determine the true success factor.

Which Split Testing Method is Best for You?

Both methods have their place and are uniquely suited for test-specific outcomes that impact inbound traffic and conversion. It all comes down to your main metric: Ask yourself, are you attempting to attract more visitors from organic search, or influence more conversions from existing traffic? 

If it’s the latter, standard A/B testing is the way to go. If it’s the former, true SEO split testing is the method you want. It’s often a bit more difficult than standard A/B testing, but it’s the only true way to test changes that impact organic search results. 

Recommended Reading: Top Metrics to Use in SEO Split Testing

Keep in mind that, like all things in organic search, SEO split tests take time since it requires that search engines crawl your test pages and reindex them comparing and re-ranking them against the unchanged pages. 

This can take two weeks or more just to detect the changes on the Google SERP, even for a frequently crawled website. Then you can start to measure the user impact. Standard user A/B testing can sometimes reach statistical significance in hours or days for really busy pages or within a week or two for moderately popular sites.

Can You Run Both Tests at the Same Time?

You can, but it’s not advisable since it’s difficult to accurately measure the results of tests running against the same pages. 

Since standard user A/B testing often involves some content being hidden from search engines, this winning content may later influence organic search. Therefore, it is best to coordinate the two testing methods carefully and thoughtfully. 

Which should you do first? Again, it depends on your primary goal: increased conversion or increased traffic. 

Both, you say? We agree, and suggest you do user testing and SEO split testing on a regular basis.

Conclusion

Both methods of testing described above have their own purpose — and considerations. 

One big difference to keep in mind is the time it takes to measure the results. Again, user A/B testing results can be measured in a day or within several days, depending on traffic levels. 

Since SEO testing takes time for search engines to crawl, index, and re-rank the test pages, expect this type of test to last 2 weeks or longer, depending on traffic levels.

It’s also important to remember that if you don’t hide the user testing from search engines, it could have negative consequences like lower ranking or even a search engine penalty, both of which result in reduced traffic.

Schedule a demo and we will show you how we use our platform for SEO split testing.

<<Editor's note: this post was originally published in December 2021 and has since been update.>>