A/B Testing Email Campaigns

Campaign Deputy supports A/B Testing for email campaigns. Each different email is referred to as a Variation or Variant, starting with Variant A. For optimal results, we recommend an email list size of at least 2,000 deliverable emails. For smaller lists, consider using the Send to All option.

To enable A/B Testing, slide the A/B Testing slider to the right. The current From, Subject, and Body fields will be copied to Variant A. Disabling A/B Testing will result in the loss of any changes made to the Variants.

New Variants can be added by clicking the Add Variant button. Variants are labeled alphabetically starting with A. Note that Variant A cannot be removed.

A minimum email list size of 1,000 is recommended before using A/B Testing.

• Send to All: Evenly distributes emails across all variants. For example, if there are 101 emails and 2 variants, a variant will be randomly chosen for the 101st email.

• Test Open Rate: Determines the winner based on unique open rates, excluding proxy-based opens. The calculation includes delivered emails and the open rate per variation.

• Test Click Rate: Determines the winner based on unique click rates, excluding clicks on the Unsubscribe link. The calculation includes delivered emails and the click rate per variant.

After the set Test Duration, the winner is calculated. The timing may not be exact as the system attempts to recalculate later for a higher confidence score. The duration starts after all emails have been sent, and the system may delay calculating the winner if there are several email opens or clicks arriving just before the calculation.

A 95% confidence score is used to determine the winner. If the target list size is not large enough or the open/click rates are not significant enough, instructions to pick a variant manually will be emailed.

Suppose there is an email list of 4,000 emails and a Test Size of 20%. Variant A delivers 160 emails, while Variant B delivers 195 emails due to bounces or full mailboxes.

• Variant A: 48 opens out of 130 delivered emails (36% open rate)

• Variant B: 50 opens out of 195 delivered emails (25% open rate)

Variant A is the winner as it has a higher open rate compared to Variant B.