A/B Testing and the Benefits of an Experimentation Culture by Wyatt Jenkins.

Modern tech companies have figured out that data is their product. Whether you sell a service, a product, or content, what you really do is create value for your customer base — and every interaction with your product is a measurable amount of value. But the best data-driven companies don’t just passively store and analyze data, they actively generate actionable data by running experiments. The secret to getting value from data is testing, and if you’re looking to grow your online business, implementing well-executed, consistent A/B testing is a necessity.

At Shutterstock, where I work, we test everything: copy and link colors, relevance algorithms that rank our search results, language-detection functions, usability in downloading, pricing, video-playback design, and anything else you can see on our site (plus a lot you can’t see).

Shutterstock is the world’s largest creative marketplace, serving photography, illustrations, and video to more than 750,000 customers. And those customers have heavy image needs; we serve over three downloads per second. That’s a ton of data.

This means that we know more about our customers, statistically, than anyone else in our market. It also means that we can run more experiments with statistical significance faster than businesses with less user data. It’s one of our most important competitive advantages.

A/B Testing in Action
Search results are among the highest-trafficked pages on our site. A few years back, we started experimenting with a mosaic-display search-results page in our Labs area — an experimentation platform we use to try things quickly and get user feedback. In qualitative testing, customers really liked the design of the mosaic search grid, so we A/B tested it within the core Shutterstock experience. For those unfamiliar with A/B testing, it involves, at its core, showing different user experiences to different users to measure the impact of those differences. You can read more about it here.

Here are some of the details of the experiment, and what we learned: