Have you ever had the feeling that you’re not quite living up to your potential?

You know there’s more you can do. You may even have a couple of ideas about which direction to go in. But you’re just not sure which one to follow. Path A looks like it leads to the promised land, but so does path B.

After a lot of contemplation, you finally make your decision, but are left wondering if you really made the right choice.

It’s similar in software development. The difference is that in software development you can find out if you made the right choice. Better yet, you can find out in a very cost effective and low risk way.

It’s all to do with something called A/B Testing.

Why Use an A/B Test

As with humans, every piece of software has so much potential to be improved upon. There are so many choices you can make. Changing the color of this element, increasing the size of that element, changing the call to action, or one of many other design choices.

Trying to work out which is the right choice can be enough to drive you mad.

These choices may even seem trivial. But every choice you make can have far reaching consequences.

A simple change of color or text used in a call to action button can result in more signups, which means more people to market to, more sales and an increase in your bottom line. Repeat this for other elements and it’s easy to see what at first seemed trivial is anything but.

How an A/B Test Works

Show 50% of visitors option A, and 50% of visitors option B. Let the test run for some time to get a statistically significant difference. Then see which one wins by whatever metric you want to test by, like an increase in sign-ups.

Simple enough.

It isn’t just individual elements that can be tested against each other, but also different design directions too. Say, you want to test the difference between a page that is bold and impactful against one that is unassuming yet colorful.

The same principle applies.

You’re not just limited to testing one page or element against the other either. You can simultaneously test three, four or however many elements and pages you like. The A/B test becomes an A/B/C/D test and so on.

But what about if you want to test more variables and see which combination gives the best results.

You can use something called a Multivariate Test.

Multivariate Testing

Multivariate testing can be very powerful. The same principle applies but this time you change up more than just an individual variable.

There’s no better example to show the power of this type of testing than by taking you back to the 2008 US presidential election.

No doubt you remember the 2008 election. It was an historic affair with the USA electing its first black president, Barack Obama.

Obama’s campaign was like any other in that it needed volunteers and donations to sustain itself. And one of the ways the campaign team was looking to drum up volunteers and solicit donations was through its website.

The website’s primary objective was to gather email addresses and then email those who signed up to encourage them to volunteer and donate.

In the hope of increasing signups they decided to test different media and buttons against each other.


They tested three images:

  • Obama surrounded by supporters and campaign banners
  • Obama looking very presidential.
  • Obama with his family

As Obama was charismatic and an excellent orator, they also tested three videos.

Various CTAs were also tested. Each button stated one of the following :

  • Sign-up
  • Join Us Now
  • Learn More
  • Sign-up now.

Before asking you which combination you think performed best. Which form of media do you think performed best, the images or videos?

If you said the videos, you wouldn’t be alone.

The campaign team themselves “heavily favored” the videos and there was one in particular they thought would be a runaway winner. But to their surprise, the images outperformed every single video.

How about the call to action?

I don’t know about you, but I would’ve assumed it would be “Join Us Now.” After all, politics and election campaigns are always impassioned affairs with every person taking a side and standing firm. “Join Us Now” delivers more impact and delivers that sort of tone.

In fact, the combination of the Family image and “Learn More” call to action was the winner.


Signup increased from 8.26% to 11.6% – which is a 40.6% improvement. This meant 10 million instead of 7 million emails were collected – a 3 million difference.

Existing data told them that approximately 10% who signed up would volunteer. This meant the number of volunteers increased by 300,000.

The average donation per signup was $21. The 3 million additional signups were therefore worth an additional $60 million in donations.

If the campaign team went with their assumptions without hard data to back it up, it clearly would have been a huge mistake.

There would have been 300,000 fewer volunteers and $60 million less in the kitty.

The Benefits of A/B and Multivariate Testing

This experiment highlights three clear benefits of A/B and Multivariate testing.

Data Driven Design

The decision about which image to use was based on hard data. Data that can’t be argued with.

A/B and Multivariate testing is therefore the ultimate form of data driven design where every decision is backed up by statistical data.

Cost Effective

Election campaigns don’t come cheap. Millions can be spent in any one area in an attempt to achieve small gains. But in this case, all it took to increase volunteers by 300,000 and donations by $60 million was a few simple changes to a website and running some software to track results. Talk about a high ROI!

A/B and Multivariate testing is therefore very cost effective.

Low Risk

No drastic changes or putting the campaign at risk was necessary. If the changes performed worse, they could easily be reverted with very little damage done.

A/B and Multivariate testing is therefore a low risk endeavor.

Validates or Invalidates Assumptions

The campaign team had their assumptions which changes would work best. You may have even shared those assumptions. It turns out these were wrong.

A/B and Multivariate testing therefore either validates or invalidation assumptions, so you can be sure the right choice is being made.

As we’ve said before:  In UX, every time you make an assumption, you are playing a dangerous game.