Each version of your split test campaign is analysed to determine which is performing the best. The remaining balance of your contacts for the send are then sent that winning version. This allows you to test and maximise your campaign performance using the best performing variables.
You're able to choose the metric by which a split test winner is judged - opens, clicks or ROI.
To be able to choose the ROI metric you need to already have ROI tracking set up for your account. ROI is not available as an option for split test metrics if your account uses Advanced revenue attribution.
Before you start
Things you need to know:
You should be familiar with the basics of creating and sending email campaigns before making use of split testing.
Create a split test
Split test settings are configured during the Create step of the campaign building process.
To create a split test email campaign:
Go to Campaigns > Email and either select NEW CAMPAIGN to create a brand new email campaign, or select a campaign name to edit an existing campaign. If you choose to edit an existing campaign, it may open on the Content step of the campaign creation process. Select Create from the breadcrumb menu in the top right to configure your split test settings.
Select SPLIT TEST SETTINGS.
In the Split test settings side panel, toggle the split test features you want to use to ENABLE.
Subject lines - Critical for contact engagement.
Friendly name - Find out which from name your contacts are most responsive to.
Creative - Identify the most effective design or content for your campaign.
From address - See which originating email address your contacts are most responsive to.
The From address option only appears if you have more than one from addresses configured on your account.
Select APPLY to save your settings and create a split test email.
Use split test emails
With split testing enabled, two different versions of your campaign are available by default, accessible by tabs in the top left. Select the + symbol to add more versions, up to a maximum of ten. Select the X symbol to delete a version.
For split tests based on subject line, friendly name or from address, you see the version tabs on the Create step. For creative, you see the version tabs on the Content step.
You can only add versions on the Create step, not on the Content step.
Select a tab to see that campaign version. Enter your content for each version, then select SAVE or SAVE & CONTINUE.
Configure and send a split test campaign
Once you’ve set which campaign elements you want to split test, you need to configure your split test options prior to sending. These options comprise of three key components:
The sample percentage of your selected contacts that you'd like to send the split test to.
The number of hours over which you'd like to measure responses.
The metric you'd like to use to judge a winning version: opens, clicks, or ROI data.
Split test options
You can set up your split test options during the Contacts step of the campaign creation process.
On the Select contacts and schedule campaign screen, you first need to select the audience to send your campaign to. This is the lists, segments or marketing preferences who you want to be sent the campaign.
Next, you need to set the send time.
Select Immediately if you want the campaign to be sent as soon as you have finished setting it up, or At a scheduled date and time for it to be sent in the future at a date and time you specify.
Now you need to set your split test options:
Percentage of audience to split test
Enter a percentage relating to how many of your contacts you wish to use for the split test. The default setting is 5%. The number of contacts selected here is split across the variations of the campaign you're testing (for example if you have selected 10% and have a split test campaign A and a split test campaign B, then 5% are sent A and 5% are sent B). The remaining 90% of your selected audience are sent the winning campaign version once the testing period is finished
Split test length (hours)
Enter the amount of hours you want the responses measured for. The default setting is five hours. This means that after five hours the rest of our contacts are sent the strongest performing campaign; the winner of the split test.
Measure
Select whether you wish to measure the winning campaign based on opens, clicks or ROI when recording the responses.
We advise using clicks, or ROI if you have it configured, as the test metric rather than opens. This is because the way in which some operating systems’ mail clients handle opens data can affect the accuracy of this metric.
Split tests and external dynamic content
If your split test campaign is making use of external dynamic content, then this may cause delays in completing the split test send in the time limit you've configured.
This is because each external dynamic content block performs an HTTP request, which can generally be slow to make. If you have a lot of external dynamic content blocks in your campaign then this could significantly reduce its send speed, and this delay may exceed your configured time delay.
The ROI metric
If you select the ROI metric you're prompted to select one of your markers by expanding the drop-down menu.
Which marker you choose, and correspondingly its data type (text, number, date or yes/no) dictates your next set of options.
For instance, a marker with a number data type provides you with a choice of hits, total and average. This is because a marker with a number data type isn't only hit upon by a visitor to your website page; the numeric value included with the marker means a total and average can be worked out for each split test version.
In the case of CheckOutAmount, a winner can be decided upon the total amount of spend for all customers and the average spend across all customers.
Anything other than a number data type for a marker gives the default choice of the hits metric only, as nothing further can be calculated for these data types.
When selecting a different metric option, a box appears providing further information on what the metric means and what it's useful for.
Save and send your split test
When you’re happy with your options, select SAVE & CONTINUE.
The campaign settings are displayed for you to review. Each set of variables are shown as a different summary tab.
Once you’re ready to send, or schedule your send, select either SEND CAMPAIGN IMMEDIATELY or CONFIRM SCHEDULED CAMPAIGN depending on the option you chose for the campaign send time.
A confirmation message is displayed. Select CONFIRM.
The campaign is moved to the Outbox for processing, and this is shown by the indicator displayed under the Status column. Once the send is completed, the campaign moves to the Sent tab.
Split test reporting
After your split test has been sent, you can view the campaign report to see statistics about the send, and how each split test version performed. To view a campaign report:
Go to Campaigns > Email > Sent tab, or Analytics > Reports.
Select the campaign name.
If your split test is in the outbox, you can see how long remains before your split testing period has finished.
On the campaign report, you see a section titled Split test results, which by default displays the winning split test. The selected metric is displayed as a percentage, along with the Subject line, From address and Friendly from name. Select + SHOW LOSING SPLITS to display the same information for the other versions of the campaign.
Select VIEW DESIGN to open the campaign preview window displaying the relevant campaign version.
In the event of a tie…
In a situation where we can’t determine a winner based on the metric selected, version A is sent to the remainder of your contacts.
This can happen if there is no engagement with your campaign during the initial test period. For example, if you have selected clicks as your metric, but no contacts clicked during the test period. When this happens, filtering the report by the winning version might show engagement, but the filter shows data for both the test period, and the period following the winning version being sent.
Filter by split test variation
Select FILTER: SHOWING ALL, then, in the Filter report side panel, select Who received the selected split test variation.
Select SELECT SPLIT TEST, and select the radio button for the campaign version you want to filter by.
Select APPLY. The report refreshes and updates to display the relevant statistics.