6 Common A/B App Testing Mistakes You Probably Don’t Realize You’re Making by Logan Merrick

app testing mistakes

Logan MerrickLogan Merrick is the Co-founder and Director of Buzinga App Development, Australian leaders in mobile product design. Download his free Marketer’s Guide To App Store Optimisation for data-backed steps to take your app to #1 in 2016. Today he shares with us a few a/b app testing mistakes that some of us may not even realise we’re making!


6 Common A/B App Testing Mistakes

With A/B testing for apps being such a new industry (just like apps themselves), developers are falling into the same traps again and again. Even a small oversight can ruin the reliability of your data, which informs all major decisions to do with your app page. Good news: They’re all easily avoided. Here are the most common A/B testing mistakes you need to look out for.

Not tracking the entire funnel

Take this scenario: You A/B test your app’s description copy and find that downloads increase. At first glance, this might look like a win, and it probably is!

But if you stop your tracking there, you won’t find out what happened to those new installs you attributed to that app description. What if you find that 80% of them uninstalled your app within a day of downloading? That’s not quite as positive an impact as you first thought. Perhaps their expectation wasn’t met, and you need to go back to the drawing board.

If you stop your tracking too early in the funnel, you only get a shallow picture of the overall effect of your change.

The further you can drill down, the more accurate and insightful your analysis will be.

Not running tests for long enough

app testing mistakes

I know, it can seem like a waste of time continuing to run a split test when the data is showing that there is far and away a clear ‘winner’.

But it really is true that the more data you have, the more reliably you can draw conclusions from it. Especially if the results are going to inform a decision that will cost you a lot of money!

So how long should you run tests for?

Some experts suggest running your experiments until you’ve tracked thousands of conversion events.

But, in truth, it’s not so much the number of conversions that matters (although low numbers are also not recommend); it’s more so if the time frame of the test is long enough to capture variations on your page.

Types of variations:

  • Do you get different conversion patterns during the day vs night?
  • Weekdays vs weekends?
  • Start of the month vs end of the month?

If you can, allow for a few cycles of variability in order to normalise your data.

Not taking into account real life

Pay close attention to any extraneous factors that might produce illusory test results, like public holidays or different time zones of your app store page visitors.

Because external factors like this are very common and often hard to identify the root cause (haven’t we all seen a huge spike in downloads and have no idea where it came from?), it can be a good idea to run the same test twice.

Some red flags to look out for:

  • The result of a test produced highly unexpected results
  • Individual conversions were very diverse. The mean conversion rate may not be an accurate representation of the overall impact of the change.

If you have the resources, when in doubt, just run a follow-up test.

Doing A/B testing even when you have zero traffic to your app page

Kind of a no brainer but it is worth stating: If you only have 10 visitors to your page each month, the results of a 1-month A/B test won’t be statistically significant.

I also wouldn’t advocate just running the same tests for months and months on end until you DO get some traffic.This is just a waste of time and money.

Instead, just bite the bullet and make the change based on best practices. At least a bonus of not having many leads is that you don’t risk much when making a drastic change!

Not testing based on a clearly defined hypothesis

app testing mistakes

You know what’s expensive? Desperately throwing tests against a wall until something sticks.

If you A/B test every random idea you have, it’s going to add up to a lot of wasted time and potential leads.

A hypothesis is proposed statement made on the basis of limited evidence that can be proved or disproved. It is used as a starting point for further investigation.

What seperates a good from a bad hypothesis is in the initial research that informed the hypothesis.

Start with examining where the problems lie on your app store page.

For example:

If you’re not getting much traffic, it’s more likely to do with your app icon than it is with your description, which people can only read once they click through to your full page.

This would inform your hypothesis that an icon with more vivid colours would increase traffic over your current minimalist icon. You can then A/B test this hypothesis quite easily.

On the other hand, if you’re getting a lot of traffic but not seeing a follow through in conversions, the problem is likely not to do with your icon. I’d be starting with your app screenshots or description first.

Not being aware of selection bias

Selection bias occurs when we wrongly assume that some portion of your traffic represents all of your traffic.

You’re probably driving traffic to your app page from a range of mediums – your email list, social media or blog content, for example.

The problem with this is that they are likely to exhibit very different behaviours. NOT ALL TRAFFIC IS EQUAL. A visitor who came from your email list or who follows you on social media is of course more likely to convert than someone who found you through a random search term on Google. They have already indicated that they like you!

Consider segmenting your users into different buckets and A/B testing based on demographics like age and gender, or behavioural characteristics like acquisition channel, device type and new/returning visitors.

Where you get really interesting results from A/B testing is when you can figure out what segments converted the highest.

This will reveal where you need to funnel more of your budget into and which segments you need to nurture more.

For example, you might find that you’re getting the most downloads from a Facebook ad, but the downloads you acquire from organic search on the app store are much more likely to become active, engaged users. This harks back to point number 1: Remember to track the entire funnel! And now we’ve gone full circle.

A/B testing doesn’t have to be rocket science, but there are a lot of easy traps to fall into. Talk to your A/B testing provider like TestNest about how they minimise these common mistakes and biases.

What are some A/B app testing mistakes you have made that could have been avoided?

  • Marcus Miller

    One mistake every A/B tester do is relying on same techniques. Your testing depends on your industry and customer base. Figure out what you want to test and how you want to test. Marcus Miller// Manager, http://www.enterprisemonkey.com.au