Want to Make Better Decisions? Start Experimenting

Learned a lot lending an editorial hand here:

MIT Sloan Management Review, Summer, 2020

by Michael Luca and Max H. Bazerman



Image courtesy of Ken Orvidas/theispot.com

Suppose you work on Google’s advertising team and need to decide whether ads should have a blue background or a yellow background. You think that yellow would attract the most clicks; your colleague thinks that blue is better. How do you make the decision?

In Google’s early days, the two of you might have debated the issue until someone caved or you both agreed to kick the decision up to the boss. But ultimately, it dawned on leaders throughout Google that many of these debates and decisions were unnecessary.

“We don’t want high-level executives discussing whether a blue background or a yellow background will lead to more ad clicks,” Hal Varian, Google’s chief economist, told us. “Why debate this point, since we can simply run an experiment to find out?”

Varian worked with the team that developed Google’s systematic approach to experimentation. The company now runs experiments at an extraordinary scale — more than 10,000 per year. The results of these experiments inform managerial decisions in a variety of contexts, ranging from advertising sales to search engine parameters.

More broadly, an experimental mindset has permeated much of the tech sector and is spreading beyond that. These days, most major tech companies, such as Amazon, Facebook, Uber, and Yelp, wouldn’t make an important change to its platforms without running experiments to understand how it might influence user behavior. Some traditional businesses, such as Campbell Soup Co., have been dipping their toes into experiments for decades. And many more are ramping up their efforts in experimentation as they undergo digital transformations. In a dramatic departure from its historic role as an esoteric tool for academic research, the randomized controlled experiment has gone mainstream. Startups, international conglomerates, and government agencies alike have a new tool to test ideas and understand the impact of the products and services they are providing. Read the rest here.

Making experiments pay

strategy+business, June 2, 2020

by Theodore Kinni




Illustration by AndSim

The current pandemic is a dramatic case study in why experiments matter. How else are we to know which tests accurately identify COVID-19 infections and their telltale antibodies, whether and how well drugs already at our disposal mitigate the suffering inflicted by the coronavirus, or whether the vaccines currently in development will actually protect us from it? Randomized controlled trials are the only way to answer these questions short of playing Russian roulette with the lives of large numbers of people.

Experiments have been around for a long time. In The Power of Experiments, an introductory paean to the benefits of experimentation for corporate and government decision making, Harvard Business School professors Michael Luca and Max Bazerman peg the first recorded experiment to the reign of King Nebuchadnezzar in Babylonia circa 600 BC.

As the Old Testament story goes, Nebuchadnezzar attacked Jerusalem and took a group of young Israelites, including one named Daniel, as servants. He ordered that they be acculturated for three years, in part by being fed the same food as the royal court, which the Israelites considered “ritually unclean.” But Daniel proposed that he and three other prisoners be fed only vegetables and water for 10 days, at which point a guard would compare their health to that of the rest of the prisoners, who would have been fed the local fare. At the end of the experiment, the guard judged the vegetarians healthier, and Daniel and his three comrades were able to continue eating the less objectionable diet.

About 2,600 years later, Luca and Bazerman write, “We are in the early days of the age of experiments.” By this, they mean that experiments have spread far beyond the boundaries of science labs and medical trials. In 2018, Google ran more than 10,000 experiments. Amazon, Facebook, Uber, Yelp, and TripAdvisor run thousands of experiments per year. As these names indicate, technology companies, especially those with digital platforms, are at the forefront of corporate experimentation. That’s because it is relatively easy to run experiments on their large, captive audiences. Read the rest here.