Revisiting the reasons for A/B testing

Paras Chopra defends A/B testing from what he feels are some unjust criticisms.

He makes some good points and I ‘ve copied over his summary here:

To reiterate the lessons learned from the three arguments above:

  • Because you can never achieve the global minima, embrace the local minima. Testing trivial changes takes a few minutes, but the potential outcome is far greater than the cost of those minutes.
  • Constantly explore the best ways to increase your conversion rate by performing both trivial tests and radical redesign tests at regular intervals.
  • A/B testing is a tool and does not kill your imagination (in fact, you need your imagination most when designing variations).
  • Lastly, don’t feel guilty about performing A/B testing.

The first point (on local vs global minima) is simply an argument about statistical theory. In English, Mr Chopra is simply saying that just because “better isn’t perfection”, it doesn’t mean you shouldn’t strive to do better anyway (and also because better will tend towards perfection).

The second and third points are simply unfounded arguments between science and art; that stone cold measurability will stifle creativity. That the process of optimisation or continuous iterations are in conflict with radical redesigns and reimaginations. These are just not true.

A/B testing is just a tool. Like any tool, it has benefits as well as flaws. Like any tool, its usefulness is dependant on the carpenter. And like any carpenter, understanding the strengths and weaknesses of your tool only makes you a better carpenter.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: