A/B Testing: Your Shortcut to UX Excellence

Imagine this: PaySprint, an exciting new fintech app, wants to introduce a new feature - a budgeting tool designed to keep your splurges in check. They have 2 variants of this feature - ‘Variant A’, which includes playful screens with fun animations, rainy day funds and whimsical piggy banks that burst into coins when users hit their savings goals. ‘Variant B’ serves as the intellectual sibling, offering a library of budgeting articles, interactive infographics, and bite-sized finance podcasts. How did PaySprint decide the winner? With an exciting face-off of A/B testing. When the experiment was done and the data was analysed, it was ‘Variant A’ that won users’ hearts (and wallets).

So what is A/B testing??

A/B testing, also known as split testing, is a way to compare two different versions of a webpage, mobile screen, feature or other user experience to see which performs better with your target users. In the A/B testing arena, you take your existing design ('Variant A') and square it against a modified version ('Variant B').

It works like this: half your users see Variant A, and the other half see Variant B. Their interactions are tracked, from clicks and scrolls to sign-ups and purchases. After a set period of time, you analyse the data to see which version achieved the most success based on your predefined objectives. A/B testing is more than just a game of guesswork—it's a powerful tool driven by data and user behaviour and it gives users a voice in designing their own experience.

‍

When you need to do A/B testing

So when exactly in your UX process would you want to implement A/B testing? You want to do this when your app has a stable tribe of users and you're aiming to jazz things up, not completely reinvent the wheel. Think of it as your guide in the realm of "Should we go for a blue 'Sign-Up' button or stick with the red?" or "Is 'Join the party' more enticing than 'Sign up now'?" A/B testing is your digital detective for these mysteries. Consider doing A/B testing when you're questioning specific design elements.

Steps for running A/B testing

‍

1. Draft a hypothesis

The first crucial step in your A/B testing journey is to draft a hypothesis. A hypothesis, at its core, is an educated guess - a question you need to answer or an assumption you need to test. For PaySprint, we could hypothesise: “Turning our ‘Save’ button from blue to green with an animation will increase the usage of our budgeting tool.” It's an inference drawn from user feedback and behaviour patterns. A powerful hypothesis is always specific, insight-driven, and tightly linked to a measurable outcome.

‍

2. Choose the right metrics

Once you have your hypothesis down, you need to choose what success looks like for this A/B test - which means picking the right metrics.

For PaySprint, you could measure metrics such as bounce rate (how many users visited and immediately left), the time users spend within the app, or the number of new budgeting tool registrations. However, remember that your goal is king here. If we're trying to increase the usage of a particular budgeting feature, our focus metric might be the engagement rate with that feature.

‍

3. Select a feature to test

This step is the “what” of our A/B testing investigation. Remember our PaySprint hypothesis? We’re betting that changing our 'Save' button colour will increase the use of our budgeting tool. This is the feature we want to test - our button's colour. You could test virtually anything in the app, from the size and location of images, the length and content of your text, to the location and number of fields in your forms.

Once you've picked your feature to test, it's time to create a mirror image of your app. Keep in mind, to maintain a clean experiment, we’re only changing one thing at a time. So, button colour it is, no more, no less.

4. Set sample size & test duration

Step 4 is akin to planning the perfect birthday party - you need to know how many guests to invite (sample size) and how long the party should last (test duration).

Your sample size is the number of users who'll get an invite to your A/B party. Too few and you might miss the trends, too many and you could be waiting forever for the results. The sample size depends on the expected impact of your change, and there are handy calculators online to help you find this out.

The duration of your test will also be project specific. The ideal duration for an A/B testing experiment for the PaySprint app largely depends on the app's user behaviour and the volume of traffic the app receives. In general, though, it's often recommended to run an A/B test for at least two weeks. This allows for capturing user behaviours throughout an entire week, accounting for any potential variability in user engagement from weekdays to weekends.

However, it's also crucial to consider the cycle of user engagement with the app's specific features. For instance, if the hypothesis is about a feature that users typically interact with on a monthly basis (like setting up or reviewing a monthly budget), then the experiment might need to run longer to capture enough interactions for a reliable result.

It's also important to ensure that the experiment doesn't run too long as it might be subject to effects from external factors (e.g., seasonality, special promotions) which could influence user behaviour during the testing period.

All things considered, the ideal experiment duration would require a balance between capturing sufficient and accurate data while minimising the influence of external factors. Always remember to base your decision on your app's specific characteristics and the nature of the hypothesis being tested.

If you’re struggling to decide on the duration of your test, use this calculator.

‍

5. Run a test

Once you’ve followed the steps above and decided on a method of A/B testing (more on that here), you’re ready to run a test. Before you launch straight into the A/B test, it's smart to start with an A/A test

An A/A test is like an A/B test's identical twin, except you're comparing two identical versions of your app feature. This ensures your testing setup is up to scratch. If there's a significant difference between your two identical versions, then you have a problem. It could mean there's a flaw in your testing process, and it needs a bit of TLC before you kick off your A/B test.

Once you’re confident that there are no significant differences in your A/A test, it's time for the main event - the A/B test. This is where you put your hypothesis to the test. Stick to your plan, let the test run its full course, and resist the temptation to peek at the results midway. At the end of it all, you'll have some pretty exciting insights to explore.

‍

6. Analyse results and deploy changes

Once your A/B test concludes, it's time to delve into the data. Analyse the results objectively, focusing on whether the differences between your test versions significantly affect your chosen metrics. If the results indicate a clear winner that aligns with your hypothesis, confidently deploy the changes to your broader user base. However, if the results are inconclusive or the changes didn't produce the desired impact, don't hesitate to refine your hypothesis and prepare for another round of A/B testing. Each test, whether successful or not, provides valuable insights that bring you closer to enhancing your app's user experience.