What is Multivariate A/B testing?

An A/B a controlled test where you typically split your users into two different groups and expose each group to different monetization logic. You can test different monetization strategies and compare results through data analysis to understand which strategy performs better. But sometimes you don’t simply have two variations of monetization logic that you want to test. In comes Multivariate A/B testing where you can create more than two user groups to test out variations in your monetization strategy.

HyperBid’s New Multivariate A/B Testing Feature

Testing is one of the most important levers to optimize your monetization strategy. Constant optimisation requires constant testing. The ability to test out multiple variations of your strategy is now available on HyperBid and does not exist on any other mediation platform at the moment. Let’s take a closer look at what Multivariate Testing really is.

Our new Multivariate A/B test feature enables:

• Creation of multiple user groups to test features instead of just two groups.
  
  
• You can also select which placements should the test be active on. The group split that you defined will apply across the selected placements.
  
  
• View all live, completed and scheduled A/B tests in one place:
  
  
• You can now clearly view a head-to-head performance of each User Group. This is best-in-class test reporting and makes it exceptionally easy to view performance of each strategy.  You can view the performance of the variants on the overall app level as well split by placement.
  
  
• You can also Conclude your test with easier actionability. Just select which variant you want to proceed with and apply it to your preferred placements. That’s it. That’s a successful multivariate test concluded.
  
  
• You can also add A/B Test as a dimension in the Full Report now. Analysis and actionability from your A/B Tests has never been clearer.
  
  
• View your activity log with respect to your A/B Tests by selecting History Log
  
  

Getting started with your Multivariate A/B Test

Let’s get in the drivers seat for the new and improved test. Let’s create our first A/B Test with the new flow.

1. Create an A/B test
• • You can create and manage A/B tests in the Mediation Tab. Simply select A/B Testing. 
    
    
• • You can then select the +A/B Test button on the top right to get started on creating a new test.
    
    
• • To get started, select the app you want to test on and give your test a name. You can conduct and track multiple tests at the same time so make the names recognisable.
    
    
• • Next, you can select whether you want the test to be conducted across all the placements in your app or simply on your selection of placements. Select which placements you want to include or exclude in the test and you’re ready to move to the next step.
    
    
• • Now define the variants for your test. Define the User Groups that you want to test. It started with a default 50-50 split but you can add more user groups and also define the weight of each group. You can duplicate the segment and mediation parameters for user groups so that you don’t need to configure the group from scratch. This saves you the hard work of replicating the mediation conditions for every new user group that you create for the test. Once duplicated, you only have to change one or two variables to create variants that you’re looking to test. While multivariate testing allows for more than two user groups, we recommend a maximum of 4 in order to have a sound experiment.
    
    
• • Once you’re done configuring your test, you can now decide when it goes live. You could start it immediately or schedule it for a later time.
    
    
• • You can locate all your Live, Completed and Scheduled tests in the A/B Testing tab.
    
    
    
    
2. Setting up Mediation Logic for your Test Groups
• • Once the test is created, you can configure Segment and Ad Source for each Test Group on the Mediation page.
    
    
• • To make configuration changes for your user groups, simply select the App and placement that you have created the test for. Your Test groups will appear on the top.
    
    
• • If you want to make quick edits to the test groups, simply select the settings button.
    
    
• • Now depending on whether you opted to mirror mediation and segment configurations when creating the groups, you will either see the duplicated mediation configuration or a blank slate on selecting the test group.
    
    
• • Whatever your starting point, you can now start defining a different mediation logic and segment settings for your test group/s.
    
    
3. View Test Results
• • You can view the performance of your variants by going to A/B Testing and clicking the ‘Details’ button to double click on the test.
    
    
• • Here you can view head-to-head performance of your test groups. View the aggregate data of the tes on the app level as well as the performance split across placements that you have selected to conduct the test on. 
    
    
    
    
4. Concluding you Test

Concluding tests has been made easier with this update as well. You typically should choose the group with the best performance to proceed. This is of course easier to grasp with the head-to-head comparison available.

• • To conclude the test, you again go back to the A/B Testing Tab where you can view all your tests. Here you select the three dots and you can see the option ‘End Test’.
    
    
• • Here you firstly choose whether you want to apply your decision to all placements on your app property or only apply the selected auction logic to select placement/s.
    
    
• • Now if you are working with all placements, you get to choose which variant you wnat to choose for all the placements. For instance your Test Group might be the higher performing variant and hence you might want to apply its logic across placements. Once selected you only have to Submit and the auction logic will apply across all placements.
    
    
• • Now in case you want to apply different logic to different placements, you firstly have to choose that option.
    
    
• • After which, you can select a specific variant group’s logic for each placement. When viewing the reporting of variants on placement level, you may see the logic show different performance on each placement. This feature allows you to optimize your mediation strategy on a very granular level.
    
    
• • Once an A/B Test is concluded, you can view the test in the Completed tab for future reference.
    
    
    After an A/B Test is wrapped up, HyperBid SDK will still cache the strategy for a period of time. The group(s) that are not selected will generate legacy data for a while. If you'd like to view the data, you could try:
    
    • Go to full report page to query the placement data with the A/B test filter and dimension.
    • Or go to A/B Test page to view the Finished A/B Test's data.