Sidnee discusses A/B testing - How it's used, knowing what to test, and industry A/B testing tools.
In episode 5 of Digital Quarantime Sidnee discusses A/B testing, an essential step in creating fast, scalable revenue. A/B testing is the key to scaling businesses, from testing ads to landing pages to entire web experiences. If you're not doing it, you should be.
Sidnee shares her insights on the following topics:
You’re listening to Digital Quarantime, Redefining the New Normal, a podcast sharing insights about different business topics during the global pandemic. The show is hosted by Sidnee Schaefer, founder and CEO of Kingfish Digital. Thank you for listening, and enjoy the show.
* * * Intro Music * * *
Hi, everyone. Welcome to another episode of Digital Quarantime with me, Sidnee Schaefer, CEO of Kingfish Digital. Today we’re going to be discussing the benefits of A/B testing, a fascinating topic, and one that a lot of people have questions about.
The first question I usually get asked is, what is A/B testing? The answer to that is, A/B testing is a split test. It’s an experiment that you run to determine which variation of an experience performs better by presenting different versions to users at random. Then, you analyze those results. It’s the most scientific way of marketing and optimizing an experience.
You’ll find that in marketing campaigns, there’s only so much you can tweak at the ad level, where that visitor originates. And at some point, the best way to improve an experience and scale your conversion rate and drop your cost per lead or cost per sale down is to improve the experience that the user is having.
A/B testing does a lot of this measurement for you. Before, it was very manual. You’d be analyzing visitor numbers, conversion numbers; you’d be comparing between the two different versions, who got served those versions. Now, we have these fabulous tools that we can use, and a lot of them are free so that no matter what size your business is, you could be doing this. You just have to think about it at a small scale.
You absolutely don’t have to be a very large organization to have a small testing program that you run. It’s about finding the insights and thinking about what changes to the user experience might affect the user in the way that you’re looking for. To start A/B testing, you come up with a hypothesis. If you have an eCommerce site, maybe it’s “If I change this button color to blue, I’ll have more people add to the cart. That’s a very tiny example and one that a lot of people might call multivariate testing.
The difference between A/B testing and multivariate testing is nuanced. They use the same core mechanism. Multivariate testing just compares a higher number of variables and reveals more information about how the variables interact with each other. So, with an A/B test, normally, traffic to a page is split between two different versions of a design. They’re usually different versions of a design.
[3:00]
With multivariate, you’re testing smaller changes. You could be testing eight different versions of a page at a time. You could be testing different versions of a flow, as well, so there are a lot of different things you could do here. I always recommend starting with A/B testing and starting with an obvious test. So the bigger change you make to a site, the more likely you are to see a change that is relevant statistically.
So let’s get into what I mean by statistically, and the great part again is, you don’t have to be a statistics wizard to run an A/B test. You can use one of these tools that once you point one segment of users to one experience and one to the other, it’s going to tell you when it reaches confidence.
So, in statistics, this is a very brief explanation of it: they have things called confidence intervals. The higher the percentage confidence – let’s go with 98% confidence. That means that there is only a 2% chance that a result is due to chance. It’s basically saying that with 98% certainty, you can be confident in the result.
If your new variation creates more conversions than the previous one, and you’ve tested at 98% confidence, there’s only a 2% chance that there is any sort of error in that number. So, you most likely want to go with 97% and 98% confidence when you’re running a test. And that’s really the easiest way to do this is to go with the highest confidence interval.
Let’s get into how is it used in large and smaller organizations? With A/B testing, you can use this in a very large organization. The bare difference is going to be in the size of their testing program. They’re probably going to have a team of people, or they’ve hired an agency with a team of people to create tests constantly.
They have a whole testing program with hypotheses and things that might be running in conjunction with each other that don’t affect one another. When you run an A/B test, you want to try and run one at a time if you are a small company. If you’re a large organization that has thousands of eCommerce pages, you can run different tests concurrently as long as the tests don’t affect one another, so you’re not diluting your results.
You can also use A/B testing in things like advertising. A/B testing does not have to be done at the landing page or website experience level. With advertising, you can do it in Google Ads or Facebook Ads. You can run two different display campaigns against each other. You can run two different pieces of creative.
[6:00]
Facebook Ads has a fantastic testing tool for ad creative that will actually dynamically serve different images to see which ones perform better. Google Ads has something called Responsive Search Ads that do the same thing – different elements of copy against each other. You could run two different responsive search ads against each other in Google Ads, two different regular search ads.
You can do any type of test you’re looking for if you think big enough. I would say if you have an idea in your head of a test you want to run for your own company, Google it. Look up how you might execute it. If you’re concerned whether or not you can pull it off, ask your agency; ask a friend that is in the industry. I highly recommend if you’re not doing testing right now that you start doing it because that’s the thing that’s going to move the needle. You can control a lot up until that online conversion.
Now, here’s where it gets a little bit more complicated. If you have a funnel where a visitor comes in, and then they submit a contact form or a lead on your website that becomes a lead on your website, that’s only the first part of the funnel. It actually gets handed off to sales. So, what you want to do is think about: what is your visitor to lead rate?
If that’s how you’re based, then what’s your lead to marketing-qualified lead to sales-qualified lead to eventually sales? The more people you drop in the funnel in the beginning from that visitor to leads standpoint, the more your sales team is going to have to operate on. That’s how you’re going to continue to move the needle is optimizing that sale conversion rate and that lead conversion rate.
For an eCommerce company, it makes all the difference, just generally, the sales. You want to optimize that cart flow as much as possible, the way that the product descriptions are, everything. You want to think about where a user might get hung up, information that is maybe not provided. There are a lot of different things that you can think about here.
Now, the question is, how do you get ideas to test? There are a lot of different ways to do this too. The first thing you can do is look at a free tool called Google Analytics. If you already have this on your site, great. The next step is: set up your funnel on there. Go to Goals and set up the funnel. That means, what pages does somebody have to go through to get to that end conversion, whether it’s a lead or a sale?
You can see from page to page, where are people dropping off? That’s a great way to hypothesize different tests. The other one is, what pages are people bouncing on? So, they’re landing on a page, and they’re not looking at more than one page on your website. That’s another great thing to look at. Or looking at different channels and then seeing where your conversion rates are for your channels. That might also give you some ideas.
[9:00]
Secondarily, visitor tracking. This is something that you should definitely be doing that I highly recommend to my clients. That’s using a tool like FullStory or Lucky Orange. You can find a tool for any budget at this point to track visitors. What this means is – and this gets creepy. I’m going to tell you right now, this gets creepy, but this is happening to you right now when you browse on those websites.
It’s called session recording software. It’s kind of like a DVR. What happens is, when you enter into a session on a website, it starts recording your behavior. Somebody like me or a UX person or a designer is probably looking at how you’re interacting with the website. Now, when you look at enough of these, you start to see patterns. You start to see things like what they call rage clicks in FullStory, where somebody is having trouble with a feature, so that’s something you have to fix.
You might find areas where they expect something to be clickable. This is another place where you could hypothesize that if I make this area clickable, maybe I’m going to increase conversion rate somewhere else. So you’re going to drive a lot of insight from looking at a user that interacted with your website just as you would from visualizing a funnel. But I think you get more information from actually watching somebody operate.
The third thing is user experience testing. This is a little bit more qualitative. This is what user experience designers do. They go in, and they talk to people individually. They walk them through a test version of an experience. This, again, is a lot less quantitative, but it still yields a lot of potential for creating hypotheses for testing. These are some areas that I would start in. If you’re a small business just getting into this or even a larger business that’s never done this, got to Google Analytics and set up some session recordings.
Let’s talk about the benefits of A/B testing. I think you have gathered some of this from what we’ve been talking about. But what we’re really looking at is reduced bounce rates, which improves user experience. People are interacting more with your website, more pages per session. If a user is having a good experience, they’re more likely to take your desired action.
Then, the next one is improved conversion rates. A conversion could be anything for you. It’s a high-value action, lead, purchase, sale, however you want to call it, that’s what this does. You can only optimize so much at the beginning of an experience. That’s why we have to optimize the beginning, middle, and the end so that the user does what we want them to do.
And then, better ROI overall. You could scale your business immensely. They call this growth marketing. That’s how they came up with things like viral loops, which are essentially referral marketing. It’s thinking about marketing in a different way – getting into a user’s mind and optimizing for what you think would improve their experience.
[12:00]
That’s what this is. It’s a whole new level of marketing. It’s kind of a marriage between data and marketing more than anything else, and you don’t have to be a data analyst to do optimization like this. You just have to be creative, and you need to know how to track.
Speaking of tracking, what are some of the best tools? I use Google Optimize a lot. It’s a free tool. It’s got its limitations, but Google Optimize natively integrates with Google Analytics and Google Ads. If you’re running Google Ads, you can only show a new version of experience to a Google Ads visitor if you wanted to. Google Optimize acts directly to your Google Analytics. You can see results of tests in there.
[Sidnee, I had already linked Google Analytics and Google Ads earlier, so should these two be unlinked for SEO? Or will it help readers with all three linked in this paragraph?]
I think your next level up is Visual Website Optimizer. I’ve had a lot of problems with this in the past. I don’t love their Wysiwyg editor. You call it the Wysiwyg editor; it sounds ridiculous, but it’s called What You See Is What You Get. That’s how it’s said without the abbreviations. A lot of these have those. Google Optimize has one. It’s a little more complex. It requires a little bit more development, but if you’re just doing A/B testing, Google Optimize is great. You can direct 50% of your users to one page and 50% to the other.
The next level up after VWO is Optimizely. It’s a fantastic tool, but you’re looking at probably five figures for that one a year. If you’re a medium-sized business and you have enough visitors, then this is a great tool for you. That’s my favorite. Then when you get up into enterprise, you’re up to Adobe Test & Target. [Sidnee, is Test & Target the Bounteous URL?]
That’s a major investment, which is why Google Optimize is good. It’s a good solution for a lot of different people. At some point, people outgrow Google Analytics, and that’s when you get into these Adobe suites, but I don’t think it’s necessary. It’s something to think about and do vendor comparisons about, but I don’t know that that’s absolutely necessary for a lot of different people.
Finally, I want to talk about an example of how you might use this in advertising. I always think this is a really good example because most people understand it. If we talk about using this in Google Advertising, you could be doing A/B testing at the copy level, so changing messaging. You could be doing it at
[15:00]
Career defining moments for me have come from optimizing landing pages and doing something called Dynamic Keyword insertion, which we’ll talk about next time. This is a gamechanger. Being able to change the copy on a page to match your ads, your ad groups changes the quality score. It’s huge.
I don’t want to get into detail here because what this podcast was supposed to be about was the benefits of A/B testing. I think we’ve discussed a lot about this, and it’s pretty clear what the benefits are. It’s one of those things where “Why aren’t you doing it?” There’s really nothing to lose because you can use a free tool like a Google Optimize.
My recommendation to you is if you don’t know how to set this up, my team is happy to help. I’m even happy to help you. Let’s get you set up in Google Analytics in the right way; let’s talk about how Google Optimize could make a big difference for your company. Again, if you don’t need our help, really what this podcast was supposed to do is give you some ideas and show you the importance of doing this in your company because it will scale your business.
I’ve never heard of any negatives to doing optimization as long as you’re measuring correctly and you’re not making decisions based on your gut. You can create hypotheses based on your gut but don’t make the final decision to change an experience without reaching that desired confidence interval. Thanks, again, for listening to this episode of Digital Quarantime. Today we talked about A/B testing and its importance in different sizes of businesses.
* * * Outro Music * * *
[End of Episode 5 16:48]