How can we help?

Search our help center

How to Run an A/B Test in SMSBump

Our new and improved A/B testing experience for Shopify merchants is here! Users will now be able to “pre-test” their campaigns and send their subscribers the top-performing message, which is most likely to convert.

With A/B Testing, Shopify store owners choose a test group size, number of groups to test against each other with different text messages. The winning message will be sent to the remaining subscribers outside of the test groups. The winning value, which determines the winning message, can either be the number of link clicks, orders made or revenue generated


Important: The A/B Testing feature is only available to users on SMSBump Growth plans and above.


Getting Started


Recipient and Sender

On the side app menu, click on Text Marketing and then choose your Target Audience. This is the audience that will be included in your A/B test groups and will end up receiving the winning text message. That can either be:


  • All text subscribers: Launch a large promotion or storewide sale to all of your SMS subscribers.

  • Specific customers: Send a quick thank you or a secret deal to a few of your most loyal customers.

  • Segment or list: Try to reengage a list of inactive customers with an exclusive new arrival or target a segment offering a deal based on shared product interest or geographic location.



In the From field select SMSBump if you’re on a free account, and the message will send from our shortcode. If you’re on a paid plan, tick use a custom sender and write your store’s brand name (up to 11 chars max). 

A/B Test Scheduling


You can choose whether the A/B test will launch right away or later in the field Campaign Scheduling. When you click on Later, you will be able to choose a specific date and time to run your A/B test campaign.




A very powerful setting you can use with an SMSBump paid account, is ticking the checkbox Send based on customers' timezone. Enabling it will send your message based on the recipients timezone.


Important: When running A/B testing, the scheduling refers to when the test will begin. At a later step, Test period, you will determine how long the test will run before the winning group is determined. The winning message will then be sent to the rest of your subscribers that weren’t included in the test.



Campaign Name


Choose a Campaign name that is relevant and easily recognizable. A UTM campaign name tag will automatically be generated so you can track your campaign’s success in Google Analytics. You can also view the campaign results data in SMSBump’s analytics section.


Important: If you skip giving your campaign a name, one will be auto-assigned.


Smart Sending

Smart sending helps you avoid overwhelming your SMS subscribers, by disabling campaigns from being sent to those who’ve recently received one.




If you leave Smart Sending enabled, your campaign will not send to customers that have received any text messages in the last 17 hours. You can change the time interval from SMSBump Settings. Disabling Smart Sending, will send your campaign to all subscribers.



Setting Up A/B Testing 


1. To set up your A/B testing environment, you first need to select enabled from the A/B testing dropdown.


2. Number of test sets is the number of test groups that will be created to compete against each other. 



3. Then, type in your Test Group Size. This is the percentage of your chosen target audience that will be included in the test.

Advice: Use a lower test group size, i.e. 20% to maximize conversion rates. The winning message will send out to the remaining 80% of subscribers outside of your test groups.


4. Split Factor is the determiner which decides the winning test group. It can either be set to number of link clicks, order, or revenue. The message with the greatest value generated will trigger a win-case scenario to send the winning message.



5. Test Period is the duration of the A/B test. Enter a value in minutes, hours or days to set how long the testing will run before a winning message is determined. 



Writing Your Messages


You should write a different text and offer a unique promotion for each message according to the number of test sets you set. Be aware that an effective A/B test message will test only one variable, so focus your attention solely on the copy of the text, the type of discount offered, or an engaging image/GIF. This approach will let you isolate what variable worked best. 






When writing your message, keep in mind the following:


  • Adding an image or GIF allows for a 1600 character length, while emojis reduce it to 66.

  • Only use the short-codes that are available. Pasting short codes from other campaigns or automations will not work.

  • Click the add STOP to opt out checkbox to be compliant and give subscribers the ability to unsubscribe from your SMS marketing service.



Review and Send


After you are done writing your message, click on the Review & Send Campaign button to look over the details before sending.




Make sure that you see all green ticks under Campaign Overview. This means that you have filled in all the requirements correctly, and your A/B testing campaign is ready to launch.




If you would like, click on Edit to change a setting.



Finally, you should read over the details and cost in Campaign data.




Click on the Send Now button to launch your A/B Testing text marketing campaign.


Once the winning message is determined, it will send to the remaining percentage of SMS subscribers outside of the test groups. 


After the campaign is sent, you can track how well it did in the Analytics section of the app. After a bit of A/B testing, you will soon realize that the value of sending an effective A/B testing campaign, is maximizing customer engagement through the best-crafted message. 



Related articles

View the reviews from happy Shopify merchants worldwide who succeed with SMSBump.

Build your SMS marketing strategy. Grow your text message subscriber lists. Segment your audience and automate SMS marketing messages that boost your revenue and ROI.