How do I define the winning criteria with A/B testing my broadcast text?
Once you have drafted versions in the Message tab and set the test group size, you'll need to define the winning criteria. Follow the steps below to help Numero define the winning criteria and test duration.
Step 1: In the Summary tab, scroll to Define Winning Criteria. This is where you'll set the criteria for how a winner will be chosen once the test completes. At the end of the test period, Numero will analyze the data, determine a winner, and email you the results. Here are your options on the Winning Metric:
- Contributions Count: Total number of contributions
- Total Raised: Total dollar amount raised
- Conversion Rate: Percentage of recipients who clicked the link in the text and made a contribution
- Submissions Conversion Rate: Percentage of recipients who clicked the link in the text and submitted a response
- Return on Investment: Ratio of total dollar amount raised divided by the cost of sending the text broadcast
- Click Rate: Percentage of recipients who clicked the link in your text broadcast
Step 2: After you select the Winning Metric, next to that field is where you'll input a Minimum Value (this is optional to populate). This is the minimum value for the winning metric to be considered a winner. This inclusion is beneficial for customers who require a highly specific metric value. Ex: Conversion Rate (winning metric) must be at 15% (min value).
Note: If you are utilizing currency-based metrics, the option to ignore outlier contributions is enabled by default, along with statistical significance being enabled. If statistical significance is disabled, ignore outlier contributions will also be disabled. Numero enables it by default because it produces more reliable results.
Step 3: Set the Duration (hours or minutes) that will determine how long the test is analyzed for. This estimated send time is displayed next to the input fields.
Finally, you can either leave statistical significance enabled or disabled. This will help identify a statistically significant winner. If the winner cannot be determined, Numero will still show you all the stats to help make an informed decision or winner to move forward with.
