SEO Split Testing: How to Run A Successful Test in 5 Steps - [Webinar]

Digital marketers can agree on the benefits of SEO testing: justifying projects, saving time, and earning buy-in.

Not only that — you can definitively prove which SEO strategies actually work for your site and your industry rather than trying to outsmart Google's algorithm.

Prefer video? Watch our 20-minute webinar!

Unfortunately, two roadblocks prevent tests from reaching fruition:

1. SEO testing is technically difficult and requires considerable development time.
2. You need to build a business case to justify the resources to build the test.

But a manual approach isn’t the only way to run an SEO test.

When you leverage an SEO execution platform to run the test, all of these points get addressed, allowing you to focus on the outcome of the test rather than the nuances of setting it up.

In this post, we'll walk you through how to run a successful SEO split test in just 5 steps and go over how using an SEO platform can streamline the process.

What Is SEO Split Testing?

SEO split testing is the process of changing one or more on-page elements of a webpage and comparing how the original page and variant page (or variant pages if you test multiple) drive organic traffic.

It measures how search engines react to a change and how users react to the SERP. Ultimately, the entire premise is based on the fact that any change you test has to be indexable by search engines. 

Recommended Reading: SEO Split Testing vs. Standard A/B Testing: What's the Difference?

How to Run a Successful SEO Test

We've broken down the process of running a successful SEO split test into 5 steps:

• 1. Create a Hypothesis for Your SEO Split Test
• 2. Choose a Metric to Test
• 3. Create Test and Control Groups
• 4. Deploy the SEO Split Test 
• 5. Track and Review the Results
• Bonus: Decide to Implement Further or Pull Back the Changes

All of these steps require a significant amount of time, preparation, and attention to detail…

If you run the test manually.

While we’ll explain this manual approach, we also want to show you a new way to run a successful SEO test with just a few clicks.

Don’t believe us? Let us show you!

#1. Create a Hypothesis for Your SEO Split Test

The first step to successfully run an SEO test is to create a strong hypothesis.

When creating your hypothesis, you should start by analyzing your site's SEO performance. This will help you identify potential areas of improvement and provide a benchmark that you can reference when comparing the before and after results of your tests. 

Then, determine a meaningful and measurable goal for your test. This will also involve identifying which types of pages to target during the test, which will be important later when creating your test and control buckets.

You should also make sure your hypothesis is actionable regardless of the outcome of the test. If the test is successful, is it feasible to scale the changes out?

Next, consider outside factors that could impact your hypothesis. These could include seasonality, product launches or events, market trends, or competitor actions that may skew your SEO split test results and provide misleading conclusions.

Lastly, make sure your hypothesis is clear, specific, and measurable. This step is extremely important since your hypothesis will serve as a guide for the test. 

#2. Choose a Metric to Test

Choosing the right metric for an SEO split test directly impacts the validity and applicability of your test results. ​The metric you choose should either provide insight into user behavior or allow you to measure the impact in engagement and SEO performance derived from the test.

We suggest focusing on metrics like Traffic, Clicks, CTR, and Conversions which allow you to make clear data-driven decisions.

For example, if you want to optimize for conversion rate (CRO), you might consider adjusting your call to action by adjusting its location on the page or changing the form it's presented in. 

Recommended Reading: ​SEO Testing Ideas to Increase KPIs

Rankings, on the other hand, are undoubtedly an essential part of SEO and can still be monitored as part of your overall SEO strategy, BUT they are not always the best metric to use in SEO split tests. This is primarily because they can fluctuate due to various factors including algorithm updates, competition, and user behavior.

These fluctuations can make it challenging to attribute changes in rankings solely to the modifications made during a split test.

When it comes to prioritizing what to test and when, our SEO testing framework provides a set of guidelines that will help you define and design test cases.

#3. Create Test and Control Groups

Once you know what metric you want to test, it’s time to identify the type of pages you want to test.

This could be:

1. Product Pages
2. Category Pages
3. Blog Posts
4. Etc.

Focusing on a single page type allows you to have a more controlled test and gives you a clearer analysis since you're isolating the impact on specific changes and comparing variations in engagement on that single page type. It also allows you to tailor optimizations to the unique characteristics of the specific page type. ​

For example, if you are an e-commerce site and your test involves using the count of products on the page as part of the test, this would only be possible on category pages that have product counts. ​

However, depending on the hypothesis of the test, there may be situations where testing multiple page types simultaneously or conducting broader site-wide optimizations is more appropriate (i.e. testing the brand placement in title tags).​

Once you've chosen the page type to receive the test, those pages need to be sorted into test and control groups. Having a control group helps you see what other factors could affect the outcome extraneous to the test variable.

There are a few different methods to use when breaking down a page type into test and control groups such as:​

• Randomly assign individual pages within the target page type to either the test group or control group. This method ensures that each page has an equal chance of being included in either group, reducing the risk of bias. ​
• Group pages based on historical data to ensure a balanced distribution of pages based on historical performance. ​
• If you have access to a data science team, you could use existing algorithms or develop algorithms, or use machine learning models to allocate pages.​
• Take a hybrid approach where you use a combination of the above methods. For example, you could use historical performance to remove anomalies and then use random allocation to create the groups. ​

​

How to Do This at Scale

If you don't know what pages to test, seoClarity's AI-powered SEO Split Tester will help you determine what group of pages to test on, and give you a scientifically-backed way to track results.

For example, if you want to run the test on pages tagged as “Product Pages” our technology analyzes those pages and recommends:

1. The number of pages in total to test, and
2. The statistically similar groupings of those pages to receive the change and those to act as the control.

Our SEO Split Tester can even analyze your site data to offer a hypothesis for the test as well.

(Split Test Analysis presenting correlation percentage and strength.)

#4. Deploy the SEO Split Test

Now that you have your test element, your hypothesis, and the page groups, it’s time to run the test!

For manual testing, the launch is yet another difficulty since the changes need to be implemented on a specific group of pages (i.e. the test group).

Recommended Reading: Does A/B Testing Negatively Affect SEO? 

If you’re a product website, you might choose two categories that have similar traffic patterns for your testing — but this leaves a lot to chance.

Some verticals may choose to test pages based on a location like a city or zip code, and compare the results to the performance of a similarly sized city.

How to Do This at Scale

With seoClarity’s SEO Split Tester, the implementation takes place instantly across thousands of pages.

Simply define the change, and the update takes place in real-time.

Recommended Reading: How to Run an SEO Test with seoClarity

Better yet: no more waiting for the dev team to implement your test. Because, in honesty, they probably wouldn’t implement the change for you if you couldn’t prove that the test works in the first place…it’s the chicken and the egg. 

Now you can implement test variables on your own to build out that business case.

Here’s a taste of what you’re able to instantly implement on your site: 

1. Add Optimized SEO Content
2. Update or Add Internal Links
3. Make On-Page SEO Changes
4. Implement Structured Data
5. And a whole lot more …

#5. Track and Review the Results of the Split Test

Once the test is implemented and live, it’s time to monitor the results.

A good step here is to monitor the appearance of the changes in the search results. When you see that Google has picked up on the test groups’ changes and shows them on the SERPs, it’s time to monitor (though, this can be difficult to detect manually.)

A good rule is to run the test for at least two weeks — this should give Google enough time to index the changes.

Then, monitor key performance indicators.

Track rankings and traffic for the test group and compare that to the control.

This becomes a lot easier with access to an SEO platform. Google Search Console and Google Analytics also offer good insights.

NOTE: The time period you run the test is important, too. For example, organic site traffic can change drastically during the week compared to the weekend.

When you can clearly detect a divergence between the test and control, call the test positive or negative. If a divergence never materializes, call it indeterminate.

There also needs to be a significance level so you know the results are reliable.

How to Do This at Scale

One new and improved method is using Bayesian structural time series models (like Causal Impact) and a data analyst.

Even better is Split Test Analysis report, which uses the above model without the data scientist since it automatically reports clear results.  

Once the analysis reaches statistical significance, the test is over and the Split Test Analysis report is presented.

Most A/B tests are traffic-based, but this lets you test clicks, CTR, impressions, and average position, too. 

Recommended Reading: The Difference Between SEO Split Testing and Standard A/B Testing

Split Test Analysis will accurately measure changes in organic traffic and other SEO split test metrics.

The image below reveals the results of the test (positive or negative), the statistical significance, relative effect, absolute effect, and the confidence level of the test.

Bonus Step: Decide to Implement Further or Pull Back the Changes

Depending on the results of the test, the test element can either be implemented on more pages or undone. 

If the results are positive: queue it up for the dev team for full rollout with proven ROI.

If the results are negative: remove or reiterate.

How to Do This at Scale

Just like the testing elements were implemented at scale with a few clicks in step #4, you can add, edit, or adjust the specified element depending on the results of the test.

Conclusion

SEO testing definitively shows you what works and what doesn’t in the interest of your KPIs.

Although you can do it manually, enterprise sites with thousands of pages (or more!) would benefit from automated SEO testing.

The absolute easieset way to do SEO testing is with seoClarity's SEO Split Tester. First launched in 2022, it is now enhanced with more capabilities and a step-by-step workflow, allowing you to deploy new testes in minutes and automatically validate the statistical significance.

Best of all, you don't have to wait for other teams to deploy and analyze your tests.

Testing this quickly and easily means you can create a business case for new projects based on positive test results and run way more experiments each year.

Want to learn more? Schedule a free demo today!

<<Editor's Note: This blog was originally published in June 2018 and has since been updated.>>