Multi-armed bandit tests

Why you should tests Black Friday offers

Alex Lloro - CRO Expert


Managing Director

The nights are getting shorter, and summer is already behind us, which can only mean one thing; the holiday shopping spree is finally upon us. To kick off the festive season in style, businesses will be taking part in one of the most important events in the shopping calendar- Black Friday.

The truth is that if you are running an eCommerce store, you cannot afford to ignore Black Friday.

Black Friday is one of the biggest events that retailers look forward to because it can provide you with sufficient profit to push your business through the "black times.”

In this post, we discuss why Black Friday is so important to retailers and why you should take time to test your offers before the big day. So, let us get started!

Why Is Black Friday Important to eCommerce Retailers?

Traditionally, businesses would roll out a wide range of in-store deals on everything ranging from clothes to electronics, toys, foodstuffs, etc. During Black Friday, some stores could open as early as 4:00 am to start receiving shoppers.

However, things have changed greatly in recent years. With more and more businesses moving online, the shopping frenzy has also moved online. A report by The National Retail Federation shows that 2019 was a record-breaking year for eCommerce retailers as more than 142 million people shopped online. This represented a whopping 14% growth compared to 2018.

With more and more shoppers choosing to carry out their holiday shopping online, eCommerce retailers enjoy an incredible opportunity to attract new buyers to their businesses and increase their sales volume.

Some of the leading eCommerce retailers in the country have reported up to 240% and 380% increase in their revenue on Black Friday and Cyber Monday, respectively. These are insane numbers that you cannot afford to ignore if you sell your stuff online.

As an eCommerce retailer, capturing a sizable share of the growing Black Friday market presents you with a good chance to scale and move your business to the next level.

However, you must learn how to run your offers effectively and constantly test new sales angles to achieve better results.

Why You Need to Test Your Black Friday Offers

As an eCommerce store owner, you have got a lot of things to handle. So, why should you bother with testing when you already have an offer that you think will perform well during the Black Friday bonanza?

The answer is really simple. Testing will help you determine whether your offer will be effective or not. It allows you to significantly improve your customers' shopping experience translating to better click-through rates, more conversions, revenue, fewer complaints, and greater loyalty.

If you want to understand the importance of testing, think about Amazon. How did Jeff Bezos manage to transform a simple bookstore into the largest eCommerce retailer in the world? The answer is simple; testing.

In one of his interviews, he said that:

“Experiments are key to innovation because most of them never turn out as you expect, and you learn so much in the process. If you increase the number of experiments you try from 100 to 1000, you can dramatically increase the number of innovations you produce.”

Bezos' insistence on constantly testing, experimenting, and innovating has helped Amazon remain at the top of the world for so long.

Business success comes from making your customers happy and making their shopping experience seamless and pain-free. But to identify what makes your clients happy, you need to constantly test, hypothesize, experiment, and repeat.

The most innovative companies in the world deploy constant testing because they want to find ways of making their customers happy so they can keep coming back to shop with them.

The same thing applies to your Black Friday offers. Before you launch, you need to test, test, and test. Gather as much information as possible from the tests and use it to make your offers irresistible.

A/B Testing: Why It Isn’t Ideal for Testing Black Friday Offers

A/B testing is one of the common ways marketers use to test their offers. It provides them with a relatively easy way of determining which content, design, or functionality is more successful with site visitors.

A/B testing allows you to test a variation of your entire page or specific elements on the page that may affect your target audience’s behavior. You can learn more about A/B testing here.

While this type of testing is common, there is one major problem with it. In A/B testing, the customer base is usually divided into two or more segments, with each segment served a different version of what is being tested.

At the end of the testing period, the most successful variant is deployed for the entire customer base. From this permutation, two issues come up:

First, A/B testing isn't a one-time deal. To be effective, it requires you to repeatedly test improvements until you find your best possible variation. It is a highly iterative process; with each test you run building upon the results of the previous tests.

It means that you must have plenty of time to run your A/B tests successfully. If you rush the process, you may not see any meaningful gains. Ideally, you should run your tests for at least two weeks before you start evaluating results. In some cases, you may be forced to run the split tests for months to achieve any meaningful results.

Secondly, there is the issue of traffic. To run split tests, you need to segment your traffic into different groups. Furthermore, you need a fair amount of traffic to ensure that your A/B tests are statistically valid. If your site has a low traffic volume, you will need to raise it before launching your tests.

So, given the challenges highlighted above, A/B testing may not be ideal for testing your Black Friday offers. First, you may not have sufficient time to launch the tests, gather data, and use it to determine the best version of your offer.

Furthermore, the last thing you want is splitting your traffic when Black Friday is just around the corner. You need to keep every potential customer in one place to increase your conversion rate and sales volume.

Introducing the Multi-Armed Bandit Testing

Multi-Armed Bandit (MAB) testing can solve the challenges associated with A/B testing, especially for testing Black Friday offers. This type of testing involves a statistical problem setup. We shall use the slot machine example to help you understand how multi-armed bandit tests operate.

Imagine you are in your favorite casino. There are different slot machines, each with a lever. The slot machines are the "one-armed bandits" because they are known to clean out players' bank accounts.

Now, you believe that some slot machines pay out more frequently than others, so you would love to maximize this to earn more money. Of course, your primary goal is to walk out of the casino with as much money as possible.

So, your primary task is to figure out which slot machine pays out frequently and play that machine more often than others to maximize your winnings.

While your primary goal may be to play the higher-payoff slot machine exclusively, it is impossible unless you know for certain what the exact payoff rates are in advance, which is almost impossible.

This brings us to the concept of exploration vs. exploitation. It means that you really don’t know where to start in the early stages. Therefore, you need to explore different options and exploit the best-performing slot machine to maximize your profits.

During exploration, you try random machines to establish which one delivers the best reward (pays out frequently) and use that knowledge to your advantage.

Why Does It Mean for Marketers?

The same concept can be applied in marketing during offer testing. It means that every time you test a new product feature, offer, landing page, or ad, you are taking a risk because you have spotted a potential opportunity of a reward.

A/B tests are purely exploratory: customers are randomly assigned an offer variant, and the achievable payoff is the average payoff of all the variants deployed.

In such a case, the average payoff will always be lower than the payoff of the best performing variant. It means that there is an exploitation of a higher-earning variant, which might lead to regrets.

However, MAB algorithms are designed to strike a balance between exploration and exploitation by exploring each offer variant sufficiently enough to help you identify the strongest variant and exploit it for maximum ROI.

While at it, the algorithm continues to explore the other variants to establish whether they have become stronger. The idea is to end up with the maximum ROI while still allowing the MAB algorithm to continue collecting critical data on the other variants.

So, instead of splitting your traffic and waiting for too long to establish which offer variant works best, you can use MAB tests to launch your Black Friday offers because it is ideal for short tests.

Since Black Friday bonanza mostly runs for five days, you don't want to waste the five days exploring with 50% of your traffic. By the time you learn anything from your A/B tests, it will be too late to exploit the best option.

How to Implement Multi-armed Bandit Tests

Multi-armed-bandits can be difficult to implement. To implement a multi-armed bandit test, you need to employ three engineers. A data scientist to develop a clear roadmap of what you need and do some coding, a developer to input and scale the code of what the data scientist says, and a dev-ops person.

However, hiring three specialists might be too expensive on your side.

Fortunately, some tools in the market offer an implementation of MAB tests like Optimizely or Adobe Target. Some other players are currently beta testing this option.


Black Friday is crucial to all eCommerce retailers because it drives a lot of traffic to their sites leading to a significant increase in sales volumes. Taking advantage of this shopping extravaganza and running good offers can drive your profit margins up and help you clear your stock.

However, you need to understand how to run your offers to make sure you take full advantage of the annual shopping event. Therefore, testing is crucial. While A/B tests are good, they require a significant amount of time and traffic to gather sufficient data. This might not be possible due to the limited amount of time.

Running multi-armed bandit tests presents you with the best chance of "learning” while "earning." A multi-armed bandit test will drive more traffic to the best-performing offer variation, increasing your click-through rate and conversion.

If you need help implementing them, don't hesitate to schedule a free consultation with us.