When we launched True Impact last year, we introduced the idea of control groups. In Localytics, a control group is 5% of a campaign’s audience who are not sent a message. It might seem odd to not send a message to qualified users, but control groups are critical if you want to optimize your app’s marketing strategy.
If you don’t have a control group, you must make two very risky assumptions when you evaluate your campaign’s performance:
- Sending a message can only improve an app’s performance.
- If a user receives a message and then does something valuable, the message is what drove the valuable action.
Let’s talk briefly about why these assumptions are flawed, how they can contribute to a bad marketing strategy, and how using control groups can save you from drawing the wrong conclusions about the performance of your campaigns.
Assumption #1: Messages can only improve app performance
Ironically, this assumption leads a lot of marketers to send campaigns that negatively impact app performance. It’s an especially easy trap to fall into when it comes to A/B testing. For an example of this assumption in action, let’s say that you’re testing two versions of a message and get the following results:
- Users who receive Message A open the app an average of 2.2 times in the seven days after receiving the campaign.
- Users who receive Message B open the app an average 1.8 times in the seven days after receiving the campaign.
Based on this information, it’s pretty clear that Message A is the winner and you should use this message going forward. However, what if I told you that users who received neither message opened the app an average of 3.4 times during the same 7 day period? With that new piece of information, it becomes clear that you shouldn’t send either message if your goal is to increase your audience’s average number of app opens.
This scenario is a clear example of how assuming that any message is better than no message can damage your engagement strategy. Although marketing can have a strong positive impact on your app, there is such a thing as a bad message and a bad campaign. By having a control group, you are able to avoid situations where you unknowingly use a campaign or a message that is actively damaging your app’s performance.
Assumption #2: Messages are the only things that drive changes in user behaviour
As marketers, we invest a lot of time and energy in crafting our messaging campaigns. Therefore, it’s completely understandable when we are tempted to immediately claim the credit when it appears that a campaign succeeded in influencing user behaviour. The problem is, what if we are overlooking other factors that could be contributing to the campaign’s performance?
For an example of how this can happen, let’s pretend you run marketing for a dating app:
- As part of your marketing strategy, you have a push campaign that gets sent out once a week to users who haven’t “liked” anyone recently.
- On average, 29% of recipients will “like” someone within three days of receiving the campaign.
- Despite the campaign’s decent performance, you have a suspicion that a different message could drive better results.
- One week you decide to swap in a new message and the results are staggering: 52% of users who receive that week’s message go on to “like” someone during the next three days.
On the surface, it seems as though the new message led to a massive improvement in your campaign’s performance. However, what if I told you that the new message was tested in mid-January, at a time when many people start thinking about Valentine’s Day? In fact, it turns out that more recipients would have gone on to “like” someone if you had used the original message – the drastic shift in behavior was entirely due to users worrying about being single for the upcoming holiday.
With a control group, it’s easy to identify when there are external factors at play. In that situation, you will see that changes in the control group’s behavior mirror the changes seen in users who received a message. This is important because it prevents us from concluding that a campaign is more effective than it actually is, which can negatively impact the effectiveness of our marketing strategy. For example, based on the initial results in the scenario above, we might have changed the wording of all our campaigns to match the style of a message that, in reality, wasn’t all that effective.
How Does Localytics Use Control Groups?
In Localytics, control groups are a crucial part of our True Impact campaign performance reporting. They enable us to show you, at a glance, whether or not a message drove an increase (or decrease) in user engagement, revenue, or conversion. We show this through a number called “lift.” Lift takes the work out of comparing the performance of a message to that of the control group. In the image above, lift is shown as the green and red numbers to the right of each performance metric.
When Should I Use a Control Group?
Despite the benefits of control groups, many marketers are hesitant to use them because doing so would mean that 5% of the audience won’t receive a message. This is true, but our recommendation remains to use a control group with almost every campaign.
In our opinion, it’s far better to send the right message to 95% of the audience, rather than the wrong message to 100%. For example, consider the first scenario we walked through. If you chose to not use a control group, you might be able to send a message to the entire audience, but you would be sending a bad message that damaged your relationship with the entire audience!
buy ios keyword installs
