TechSavvy: Better Management Through A/B Testing

MIT Sloan Management Review, July 14, 2016

by Theodore Kinni

Marketers and product developers love A/B testing. Want to figure out the most compelling offer for an online ad or the best design for new app? Test two of alternatives head to head and see which one people like better. Voilà!

A/B testing works so well that Instacart's vice president of product, Elliot Shmukler thinks managers should adopt it to improve their decision-making prowess, too. “There are many effective decision-making frameworks out there, but I wanted to use one that would simultaneously surface the best choice for the product while still encouraging the inherently different approaches to ideation among my product managers,” he explains.

When Shmukler’s product teams can’t resolve conflicting ideas and come to him for a final decision, he refuses to issue an edict. “Instead of giving a verdict, [I] test both theories and let data be the judge,” he says. “At first pass, this method may seem to favor the data-driven people, but it empowers each PM to push ideas forward. They learn independently rather than feeling that a decision was made for them.”

A/B testing also allows Shmukler to approve multiple product ideas, with the proviso that they be tested. “It increases experimentation, autonomy and learning throughout the organization,” he says. “Most critically, it fosters goodwill among smart — but very different — PMs who want to try out their ideas.”

For more on the benefits — and the challenges — of using A/B testing for management decisions, read Shmuckler’s interview in First Round Review. Read the rest here