Learn how to use split testing with your email campaigns.
If your campaign is using split testing to test multiple subject lines, friendly from names, creative and from addresses in order to see which work best for you, then you'll need to configure your split test options prior to sending. These options comprise of three key components:
- The sample percentage of your selected contacts that you'd like to send the split test to
- The number of hours over which you'd like to measure responses
- The metric you'd like to use to record responses by and thus judge a winner with (open, clicks, or ROI data)
To be able to use the ROI data metric, you'll need to have ROI tracking and ROI markers set up in your account prior to configuring and sending a split test campaign.
ROI is not available as an option if your account uses Advanced revenue attribution.
Configuring split test options
Step 4 - Contacts - of the campaign creation steps, which is the 'Select contacts and schedule campaign' screen, allows you to set up your split test options.
You can access the 'Select contacts and schedule campaign' screen in the following ways:
- If you've closed your campaign, select Campaigns from the navigation bar and then select the Send icon against the campaign you want to send
- If you're within the campaign creation step process (this is the case if the steps are displayed in the top right corner of your screen), select step 4, Contacts
- Alternatively, you'll naturally arrive at this screen if you keep continuing through the campaign creation steps after creating your campaign content
From the 'Select contacts and schedule campaign' screen, select one or more address books and/or segments to send the campaign to.
You'll next need to specify when you want the campaign sent.
Select Immediately if you want the campaign to go now, or Scheduled to set a date and time for it to be sent in the future.
As you've selected the option to split test different variables in your campaign, you'll need to complete the split testing configuration area of the screen.
Contact percentage to receive split test: Enter a percentage relating to how many of your contacts you wish to use for the split test. The default setting is 5%. The number of contacts selected here will be split across the variations of the campaign you're testing (for example if you have selected 10% and have a split test campaign A and a split test campaign B, then 5% will be sent A and 5% will be sent B).
Time delay: Enter the amount of hours you want the responses measured for. The default setting is five hours.
In our example, we've used five hours. This means that after five hours the rest of our contacts will be sent the strongest performing campaign, which is the winner of the split test.
If your split test campaign is making use of external dynamic content, then this may cause delays in completing the split test send in the way you've configured it here. This is because each external dynamic content block performs an HTTP request, which can generally be slow to make. If you have a lot of external dynamic content blocks in your campaign then this could significantly reduce its send speed, and thus this delay may exceed your configured time delay.
Metric to be used: Select whether you wish to measure opens, clicks or ROI markers when recording the responses. As mentioned above, the ROI metric will be available as long as you've already set up ROI tracking and ROI markers in your account.
Selecting the ROI metric
There are some additional steps when selecting the ROI metric.
You'll be prompted to select one of your markers from the dropdown menu.
Which one you choose and its data type (text, number, date or 'yes/no') will dictate your next set of marker metric options. For instance, a marker with a number data type will provide you with a choice of hits, total and average.
This is because a marker with a number data type isn't only hit upon by a visitor to your website page; the numeric value included with the marker means a total and average can be worked out for each split test versions. In the case of 'CheckOutAmount', a winner can be decided upon the total amount of spend for all customers and the average spend across all customers.
Anything other than a number data type for a marker will give the default choice of the hits metric only, as nothing further can be calculated for data types of this nature.
When selecting a different metric option, a helpful box appears providing further information on what the metric means and what it's useful for.
Saving and sending your split test
When satisfied with your configurations, select the Save & continue button.
The campaign settings are displayed for you to review. Each set of variables are shown as a different summary tab.
When ready, select either the Send campaign immediately or Confirm scheduled campaign button at the bottom of the screen, depending on what you've chosen to do previously.
A final warning confirmation is displayed.
Selecting Confirm button will execute the campaign.
The campaign is moved to the Outbox while processing occurs and this is shown by the indicator displayed under the Status column. Once completed, the campaign will appear under the Sent tab.
Split test reporting
After your split test has been sent, you can select Reporting.
If your split test is in the outbox, you'll be informed how long remains before your split testing period has finished.
Otherwise, if it has completed, the winning and losing split test campaigns and their associated metric will be displayed, expressed as a percentage. Version A of your split test is sent if there is a tie.
Filtering by split test variation
To filter on split test variations.
- Select Filter: Showing all, select Who received the selected split test variation.
- Select Select split test, and select 'Version A' or 'Version B' (or C, D, and so on, depending on how many split test versions you have) from the panel.
- Select Apply. The report will indicate that it's refreshing before updating with the relevant statistics.
You may also find the following articles useful: