A/B Testing Implementation & Analysis

Conversion rate optimization (CRO) is both an art and a science. Using the scientific approach, we implement what is called a hypothesis – a building block of an experiment. It is an educated guess that may or may not be validated. Scientists use the word whenever they propose a new theory or prediction. 

In CRO we call our theories of how a website can be optimized for better outcomes hypotheses as well. The better structured a hypothesis is, the better your chances are of achieving a more successful, more organized outcome.

Example

“I believe personalizing customer logos by region will result in a better form-fill rate because it instills confidence in the prospect.”

The Why & How Of Prioritizing Hypotheses

Coming up with an idea that you want to test isn’t the toughest part of optimization. Coming up with one that you should test often is. This is where hypothesis prioritization comes into the picture. Prioritization helps you answer “here’s what we’ll test, in this order, and here’s why.”

Choosing One & Prioritizing Your Test Ideas

While it’s good to have access to a number of prioritization frameworks and models to follow, it can also be a little overwhelming. Use a model that prioritizes tests based on Potential, Importance, and Ease. Here’s how you rate the three parameters based on a scale from 1 to 5:

Potential: How much potential does the page have? Is it one that performs badly for you right now? How much do you feel it can improve? Ask these questions and come up with an objective rating.

Importance: Narrow down that page based on its importance to your business. Traffic is valuable when it’s either costly (paid) or highly targeted. Rate wisely.

Ease: How easy is it for you and your team to go ahead and both run the test and make it live? (In case it’s successful.)

How To A/B Test Like A Pro

The core of CRO is about doing things in a process-driven, methodical way. Unlike short-sighted tactics, it is not something sporadic. But if you want your CRO program to succeed, you first need to ace how you run A/B tests. 

The best CRO practitioners in the world follow a step-by-step process that looks something like the image below. It’s the five-pronged, tried-and-true approach to A/B testing.

 

• Step 1: Research – Figure out where and why people are abandoning your website by making use of quantitative and qualitative analysis tools. If you need some ideas on how to come up with a hypothesis in the first place, check out our blog post here.
• Step 2: Hypothesis Formulation – Use the insights gleaned from the research phase to create a hypothesis aimed at improving your metrics.
• Step 3: Prioritization – Prioritize hypotheses to be tested, based on various factors such as potential impact, effort trade-off, and ease of implementation.
• Step 4: Testing – On prioritization, test the hypothesis per the type of test (A/B Test, Split URL, or Multivariate Test).
• Step 5: Learning Phase – After you’ve run the test and found a winning variation, deploy it on your website. In case of a losing variation, gather learnings.

What You Should Know While Running Your A/B Tests

Things to keep in mind throughout your optimization and testing journey. 

• No Hypothesis Is Better Than An Invalid Hypothesis 

A hypothesis forms the basis of your test, it should come informed by research from

your website’s real-time, hard data. Not having a valid hypothesis is akin to shooting in

the dark. 

• Tampering With Your Test During Any Stage Is A Strict No-No 

Including, but not limited to, looking at your test results before the test is over. It leads to 

something called a “peeking error” — and it’s a fatal mistake to make. 

• Analyzing Your A/B Test Results Is More Crucial Than Anything

Your testing efforts do not stop at just having a winner; you have to go beyond and analyze each and every test to ensure continuous learning for you and your team. 

• Testing Velocity Is Directly Proportional To Your Testing Success

If there’s anything you have to make sure that you gain from A/B testing and conversion

optimization in the long run, it’s that you have to be testing All. The. Time. The more you

test, the better your chances of success. Do not wait for the perfect test. Instead,

iteratively, get there yourself.

 

Ready for the next steps to improving your CRO? Download our latest white paper, “The Crawl, Walk, Run of Conversion Rate Optimization”. It’s a complete CRO roadmap that takes you beyond analytics and puts you in the driver’s seat.

Missed seeing the first and second blogs on our CRO whitepaper? Catch up here and here.