Paddle Billing
ProfitWell MetricsA/B testing your pricing strategy is never a good idea; instead, you should survey your customers to determine your optimal pricing strategy.
So you’ve drunk the value based pricing Kool-Aid and realized the foolhardiness of cost-plus and competitor-based pricing strategies. Now what, you ask? How do I find out exactly how much people value my product? One of the ideas that might pop into your head would be to A/B test your pricing strategy. After all, it’s effective for optimizing landing pages and seems to make sense, right? Well, sort of at least.
Unfortunately, your intuition developed from years of marketing school steeped in Google’s Website Optimizer and Optimizely is misleading you. Put simply, A/B testing for optimizing your pricing structure and determining your true price point is a terrible idea; it’s a downright awful one. Yea, that’s a pretty bold statement, so let me back that up and walk you through the reasons why you should never A/B test your pricing, along with presenting some healthy alternatives.
What is price testing?
Price testing involves comparing how your market responds to different price points for your products. Instead of using optimization tools, you let the customer vote for the best pricing with their dollars.
3 Reason Why You Should Never A/B Test Your
1. Statistical significance is rarely possible for pricing
There’s a concept in statistics called statistical significance. The idea can get pretty complicated, but at its root, you essentially need a certain amount of people in your A/B test in order to ensure your end results are legitimate and aren’t just due to random chance.
When you A/B test, each of the different scenarios you run needs a statistically significant amount of test subjects in order to make sure the results are valid. This means you need a lot of customers willing to purchase your service to even consider A/B testing your pricing, and an even greater amount when testing for multiple customer personas. For nascent businesses, as well as companies who focus on completing bigger projects for a small customer base, A/B testing is an impossible method to implement.
Think you’re in the clear with your thousands of visitors per month? Well, assume that you can somehow correctly identify each visitor or lead coming to your pricing page (big assumption). Further assume that each of those groups are large enough for statistical significance across both options you’re putting forth (even more if you’re testing more than two things - bigger assumption). You still would only be shooting yourself in the foot, because you would be tracking those visitors over the course of a month or more, when evidence suggests price sensitivity can shift dramatically in even a matter of days. Plus, if you decide to change the format of your pricing page, you will need to redo the entire A/B test to eliminate any effects from the change, which will require even more customers.
To summarize, in general, A/B testing is better suited for places that generate a lot of traffic, but don’t ask for large commitments (think landing pages and SEO).
You know your business, we know pricing
Price Intelligently's team of monetization experts work with you to combine strategy and data to solve complex business problems and accelerate your growth.
A/B testing doesn’t work: how to implement a value-based pricing strategy
The best alternative to A/B testing (and the best technique for value-based pricing in general) is to survey your customer base. This avoids all of the pitfalls of A/B testing and will also provide you with better data. Here’s why:
Surveying doesn’t require as many people as A/B testing to execute, because you can target the subsets of customer personas you need. Plus, the number of respondents you can source will be greater than those who will show up in an A/B test, because unlike A/B testing, you can survey anyone who is a decent fit for your product, not necessarily those who are coming to your site. What’s more, running these studies works for all types of businesses, even those with large customer bases. Granted, you will then of course come under fire of things like survey bias, but these things can be controlled for, simply because you have control over the length of the study and the makeup of the respondents, rather than hoping and praying the right people see the right things throughout the A/B test.
Sourcing respondents also won’t anchor your customers or upset them. You’re asking for your customers’ willingness to pay, so they won’t be psychologically locked into a preconceived price when they’re ready to buy. Of course, this means you need to survey them properly (more on this in our value based pricing post). Yet, if and when done properly, you’ll minimize anchoring and reduce the risk of customer backlash, while making sure you avoid the ethical grey line of dynamic pricing.
Finally, and most importantly, surveying actually finds your customers’ willingness to pay, which is the ultimate goal of value-based pricing. A/B testing your pricing, on the other hand, only gives you relative data, which means it is practically useless for value-based pricing.
The bottom line:
A/B testing is never the best option for determining an optimal pricing strategy for your company. The practice works great for a lot of things, but its relative nature does more damage than good when coming to the most important aspect of your business. You’re much better off just speaking and sourcing information from your prospects, leads, and current customers.
To learn more about pricing specifics, check out our Pricing Strategy ebook, or speak to one of our experts. We're here to help.
