Want to Make Better Decisions? Start Experimenting

Learned a lot lending an editorial hand here:

MIT Sloan Management Review, Summer, 2020

by Michael Luca and Max H. Bazerman



Image courtesy of Ken Orvidas/theispot.com

Suppose you work on Google’s advertising team and need to decide whether ads should have a blue background or a yellow background. You think that yellow would attract the most clicks; your colleague thinks that blue is better. How do you make the decision?

In Google’s early days, the two of you might have debated the issue until someone caved or you both agreed to kick the decision up to the boss. But ultimately, it dawned on leaders throughout Google that many of these debates and decisions were unnecessary.

“We don’t want high-level executives discussing whether a blue background or a yellow background will lead to more ad clicks,” Hal Varian, Google’s chief economist, told us. “Why debate this point, since we can simply run an experiment to find out?”

Varian worked with the team that developed Google’s systematic approach to experimentation. The company now runs experiments at an extraordinary scale — more than 10,000 per year. The results of these experiments inform managerial decisions in a variety of contexts, ranging from advertising sales to search engine parameters.

More broadly, an experimental mindset has permeated much of the tech sector and is spreading beyond that. These days, most major tech companies, such as Amazon, Facebook, Uber, and Yelp, wouldn’t make an important change to its platforms without running experiments to understand how it might influence user behavior. Some traditional businesses, such as Campbell Soup Co., have been dipping their toes into experiments for decades. And many more are ramping up their efforts in experimentation as they undergo digital transformations. In a dramatic departure from its historic role as an esoteric tool for academic research, the randomized controlled experiment has gone mainstream. Startups, international conglomerates, and government agencies alike have a new tool to test ideas and understand the impact of the products and services they are providing. Read the rest here.