What is AB Testing and Why is it Important

This article is meant to explain what AB testing is and why it’s important. I will cover its practical applications as well as how I believe it’s changing the way businesses small and large approach data. At its core, AB testing leverages data to understand how to improve a website, service, application, or e-commerce storefront. The idea of harnessing data goes far beyond increasing conversion rate, but can also invoke in its users a more “data-driven mindset” when making decisions.

What is AB Testing

AB testing is the act of splitting your web or application traffic into different “variations” that you have created. Other names for AB testing include:

  • Split testing
  • Bucket testing
  • Multi-variant testing (type of AB testing)
  • Redirect testing

The primary goal of AB testing is to determine which of these variants performs the best when analyzing certain goals, such as increasing the number of clicks, downloads, or purchases. This allows you to confidently decide on changes to your website or application based on data. Which version of the site is garnering more traffic? You can then proceed with changes knowing that they will benefit your business. If your primary focus is increasing revenue or leads, then would be your “conversion goal” that informs the focus of your AB tests.

Now you might be wondering which exact AB testing tool is going to provide you with the most seamless experience.

Let’s get into the pros and cons of the most popular AB testing tools.

AB Testing Tools

There are a variety of AB testing platforms available on the market today. In essence, they are the tools that allow you to run an experiment in which two or more variants of a webpage are shown to visitors at random. Then, statistical analysis is used to determine which variation performs better for a given conversion goal.

Your tool should tell you a couple things:

  • How much traffic has been sent into each variation of your test
  • When a test has reached “statistical significance” (confirmed the learnings)
  • How different segments reacted to the test (ex: mobile vs desktop)
  • Who is seeing the test (ex: only paid ads users)

Like any competing brands with similar products, each AB testing tool will vary in how efficiently and effectively they accomplish the above tasks. Some can provide granular levels of data, others very high level data. Some are more affordable, and some are expensive. When deciding on an AB testing tool to use, you should consider where your business, e-commerce storefront, application, or team is situated in their AB testing and conversion rate optimization (CRO) journey.

Below are some of the most popular AB testing tools and what I believe to be their most significant characteristics:

  • Google Optimize
    • Free
    • Limited goal setting
    • Best for analyzing data out of Google Analytics (seamless integration)
  • VWO
    • Mid-tier costs
    • Great goal setting capabilities
    • Additional features (heatmaps)
    • Analytics are harder to setup
  • Optimizely
    • High cost
    • Most sophisticated features
    • Analytics relatively simple to setup but not easily integrated with GA
  • Unbounce
    • Free to Paid membership options
    • Simple to use
    • Limited goal setting within the tool
    • Analytics are limited but can be setup with GA

Realistically speaking, if you and your organization are just starting out on your AB testing and CRO journey, Google Optimize might be the most feasible option. You most likely have Google Analytics installed and can easily set up Google Tag Manager.

To install Google Optimize you can reference Google’s documentation.

What to do to begin AB Testing

Now that you have installed your Google Optimize account or any other AB testing tool, the fun can finally begin. It’s time to start experimenting! But wait. Before you launch your first experiment, consider the following:

  • What is your process for AB testing?
  • What are your goals?
  • Which team members will be driving you down the AB testing road?
  • How do we compliment AB testing above and below the funnel?

Keep reading to find out exactly how to tackle each of these questions

What is your process for AB testing

A normal AB testing process will normally look like this within about a month’s time. Depending on your traffic you may be able to run a single experiment or multiple experiments within this period of time.

  1. Data review :Review the data you currently have that can guide your AB testing experiments. Look at Google Analytics to identify trends and behaviors. Leverage customer segments you’ve identified through interviews or surveys. Use everything and anything!
  2. Ideation : Based on the data, start building 10-15 ideas and prioritizing them on a ratio of “level of effort : potential ROI.” This will help you prioritize low hanging fruit and build a backlog based on data.
  3. Design : Get your design team working on methods for best incorporating these new experiments. Every member of your growth team should collaborate on the design. Gathering multiple perspectives always leads to the best experiments!
  4. Development : Build your experiment in your AB testing tool or using your website CMS.
  5. Deployment : At this point you know there are no dependencies or potential problems because you’ve worked with development, design, business stakeholders, and the product team. So now you’re ready to launch!
  6. Analyze Results : You’ve run your test and gathered plenty of sessions for your experiment. You have seen a 5% increase in your goal (conversion rate, revenue, leads). WHY!? Look into the data via Google Analytics or whichever analytics tools you’re using to measure the exact changes in your user’s experience.
  7. Ideation based on winners & losers : You now understand why user behaviors changed based on your experiment because you analyzed the results! Maybe users navigated to one of your product pages more often than others which led to the increased conversion rate. What is this indicating to you? That this best selling product needs to be highlighted front and center. From here you can develop more test ideas around this hypothesis. Boom! You have new test experiments based on your AB tests. It’s a positive feedback loop that has potential to make your site better every time you run a test.

What are our goals?

So now that you know what you need to do from steps A-Z for AB testing, let’s focus on one of the most important parts of AB testing– what you are tracking!

Your business is bound to have goals that differ from any other business’ goals, so you need to clearly define these. I like to use a tier system to rank goals based on their level of importance. For example:

  • Tier 1 Goal: Revenue, Conversion rate (Leads or Sales)
  • Tier 2 Goal: PDF Downloads, AOV
  • Tier 3 Goals: Videos watched, Instagram follow

Yes, you should be tracking any goals that are relevant to your business, but keep in mind you also need to prioritize them so that when you AB test, you can clearly call out the benefits or downsides of an experiment.

Who is the Team Driving the AB Testing Roadmap

This is a key consideration when defining your AB testing roadmap. You need a team that is going to dedicate time and resources to improving your website, application, or service. Having a team that can bring diverse ideas to the table is key to gain different perspectives for AB testing experiments.

What I normally recommend for a team breakout is one major stakeholder from each product sub-team and one business stakeholder.

  • Product team member – Leader and organizer
  • Marketing team member – Top of funnel communicator
  • Engineer team member – Technical expert
  • Design team member – User focused and keeping tests on brand
  • Business stakeholder – Voice of the customer

With a team like this you can effectively understand any and all dependencies for AB tests on your website while also drawing many different perspectives.

How do we compliment AB testing above and below the funnel?

So you’ve built your team, defined your goals and processes, and now you have results showing awesome improvements. How do you keep doubling down on your success? Get more team members involved by building AB tests above and below the funnel you are testing on.
Let’s take a second to define these:

  • Above the funnel = Ads (images, videos, and posts)
  • Below the funnel = Emails, texts, post purchase communication

Every step of the way we want to be making sure your AB tests align with the above and below funnel initiatives. How could you do this?

  • Create a custom landing page that matches the ad creative above the funnel
  • Create a custom landing page that matches the email communication below the funnel

Customization like this can drastically increase your conversion rate. Very few businesses implement above and below the funnel optimizations, so when users come across this level of personalization on your website, they are sure to be impacted by it.

When you think outside the box with AB testing, the tools can most definitely support it!

Why is AB Testing Important

A/B testing allows businesses to make more educated decisions about their online presence. It provides information on how to continuously evolve and improve based on the user’s preferences. During the AB testing process you build hypotheses, understand user behaviors, and clarify what will work best.

If you adopt a mindset that includes testing everything, learning more about your users, and settling disagreements with data, it will permeate down to all parts of your business and guide you down the right path.

If you need examples of this, just look at any major company with an online presence. Facebook, Snapchat, Lionsgate, Google, Tesla, Amazon, Stripe, Twitter, Netflix, and General Motors all leverage AB testing in one way or another. They are data-backed, data-focused companies who continue their success by maintaining a data mindset.

For your business to achieve its goals, you need to use AB testing because that’s what your competitor is most likely doing.


As you begin your AB testing journey or look to improve it, remember to follow the steps laid out in this guide. Also consider how you can get more stakeholders involved to increase the investment of this critical process for your business.

Related Insights

Collab Home

CollabHome 30% opening conversion rate on launch Client: Collab is a real estate investing platform for student housing investors. Historically student housing has been a

Stream Sets

Streamsets 27% increase in leads Client: Streamsets is a SaaS software that allows you to build smart data pipelines in minutes and deploy across hybrid

Unlock New Growth

Our marketing experts, designers, developers, and copywriters have helped businesses just like yours increase
web traffic, generate leads, and drive sales. You can be the next one we help grow.

Our marketing experts, designers, developers, and copywriters have helped businesses just like yours increase web traffic, generate leads, and drive sales. You can be the next one we help grow.