Sub-Optimal A/B Testing - Why?

By: Alon Cohen Jan 14, 2018, updated: Dec 18, 2020

A/B testing is the primary tool marketing people use to optimize conversions in the digital marketing world. It’s a method to find the better converting version of a webpage or an ad.

The way A/B testing works is that you randomly present page A or page B to your website visitors and check which version of the page converts more visitors. Some tools (like https://www.optimizely.com) make that process relatively straightforward; however, it takes time to collect sufficient evidence to make a clear decision about which page performed better.

So why even bother?

The problem with webpage design is that it is hard to get it right the first time. A designer might think that the call to action button is in the way and move it to where it becomes ineffective. Color schemes and market trends affect how people perceive, understand, and operate a page.

Statistically significant results from an A/B test can help validate a webpage design assumptions and improve on them.

What can go wrong?

If, for instance, you did not assign a 50/50 impression between the A & B versions of the page, you might think that one page performs better. 

In many cases, unless one page is horrible, the other will perform pretty close to the first one, and the statistics can change week after week. You must wait a sufficiently long time, sometimes a few weeks, to get a decisive answer. 

Picking the wrong page will reduce your conversions.

Since you usually use multiple channels to target customers, changes in one medium might affect the A/B test results. Even though A/B testing could be helpful when your two pages are almost similar in performance, it usually takes a lot of work to get conclusive results, and ongoing testing is required.

So what is going on here?

Say you have used the best tools for A/B testing. You waited and got some slight statistical confirmation that one page is better. Why? Because one page was suitable for some people and the other was helpful for others.

The audience is not homogenous. When the results are close, half of the audience liked version A, and the second half liked version B. The sad product is that your bottom line stayed the same despite all your optimization efforts and patience.

Let’s say that 50% of the people like Red and half like Blue to explain it better. If you make the page Red, more red-liking people will convert. If you make the page Blue, more Blue liking people will convert. So it does not matter if the page is Red or Blue; conversions will not improve. 

Is there a solution?

Ideally, you need to have a different version of the webpage for each visitor or at least a separate page for various market segments or buyer personas. Only then could you see improvement in your total performance.

Unfortunately, I have yet to see helpful marketing tools that can tell you (the site) in real-time which version of the page to render to which user. The hope is that such a tool or an API will enable correct personalization of the webpages and drastically improve the conversion. 

It is not about knowing the visitor’s name. It is about understanding the visitor’s social or behavioral profile and displaying the correct page for each visitor’s characteristics.

In simple words, you need a way to show red-loving people the red page and the blue-loving people the blue page. This way, you can improve the total conversions and move from a "local" maximum on the optimization graph to a more "global" sales process optimization.

 

Thoughts?

Have you heard of such a tool or an API? Let me know.

Corona - Silver Lining?

By: Alon Cohen

We are social distancing and working from home. We learned that it is possible to live and work that way, maybe even work more efficiently as we spend less time on social gatherings and save the daily commute. As one who worked from home for many years, I can say that it works even better when everyone is working from home.  

From the technology perspective, people finally discovered video conferencing and learned how to use it better with the benefits and the few problems it presents. Many even learned the importance of good lighting and an impressive backdrop.

Engineered or not, the virus highlighted the upcoming international battlefield. I wish these were not needed, but I hope that we will have an increase in the research budget for developing protections against such biological weapons. As a by-product, I hope it will help find cures for health problems that plague our society.

We suddenly discovered that developing Vaccines should not take ages. Hence the cost of development “as it usually takes ten years” being the usual excuse of the pharmaceutical companies will no longer be valid. 

Pharmaceutical companies need to stop wasting time searching mostly for medications to sustain life. They rather find ways to actually cure or prevent diseases using vaccines, regenerative medicine, and genetic-medicines. 

Given the ability of those genetic technologies to cause disasters on a global scale, it might be necessary for nations to own the IP (Intellectual Property) for drugs and vaccines by financing and directing the research. 

That emphasis should be on finding cures for diseases. A cure by its nature is a less lucrative outcome for the pharmaceutical companies because once they find a cure, the disease could be vastly eradicated. 

By directing and financing biological research, governments can legally leverage that new IP in the upcoming biological warfare.

Interestingly we may have created and proven a new business model where Phramacustical companies get paid a lump sum for a product that prevents disease, a vaccine. (Operation Warp Speed)

Maybe we can extend this model, and regulate pharma so that if they want to sell us non-generic drugs, they must introduce at least one preventative or curing medication per year (as opposed to life-sustaining medicines). Once they present such a cure, the pharmaceutical companies will get paid handsomely, upfront, because it will be worthwhile for the economy to get those newly cured, healthy people, back into the workforce.

We also learned that volunteers are willing to test new vaccines to save lives to help test them faster or as they call it one day sooner. Those people are heroes in my book.

I think we learned that as a nation, we must invest more in quantum computing as it holds the future for faster material and medicine research that would otherwise take years to achieve with standard computers.

I think we realize that when the doctor says, “you contracted a virus, there is nothing we can do...”, he is probably wrong. 

With the correct focus, we can defeat viruses from Herpes to the Common Cold, maybe even Influenza. By doing so, we contribute to the economy vast amounts of money that today is being wasted on hospitalizations and lost workdays. 

We used to say that trains, then the aviation, then the Internet made the world smaller, we now realize that infectious viruses make it even smaller. A Virus does not require infrastructure or energy to travel - just abondance of people. 

Internet communication can be blocked, intercepted, manipulated, and fire-walled. The Coronavirus showed us that viruses could jump any border and if we embed data in the virus DNA we can pass information in a direct person to person manner in a way that might be intercepted but not blocked or censored.

It means that a virus can be useful as a communication chanel. A virus can be “developed” as a data carrier. We can use the virus to broadcast information with, hopefully, positive ideas, maybe even a complete version of uncensored Wikipedia, and use them to circumvent even the “Great Wall of China” or thair firewall in today's term.

###