Seeing, doing, and imagining

strategy+business, March 16, 2021

by Theodore Kinni


Photograph by Laurence Dutton

My vote for the foggiest assertion of the pandemic to date is that the U.S. has an abysmally high number of COVID-19 cases — more than 29 million as of March 8, 2021 — because of testing. “If you don’t test, you don’t have any cases,” former President Donald Trump said during a televised White House roundtable on June 15, 2020. “If we stopped testing right now, we’d have very few cases, if any.” I wonder how many people who heard that had the same first thought as I did: Correlation is not causation.

Though associations gleaned from big data drive recommendation engines and bolster corporate revenues, they have their limitations. Imagine trying to control a viral pandemic by refusing to test people for the virus.This isn’t to say that correlation — the idea that two or more things are associated in some way — isn’t valuable. Indeed, there is big money in correlation. In order to peddle subscriptions, Pandora doesn’t need to know what causes people who listen to The Grateful Dead to also listen to Phish. To bulk up its sales, Amazon doesn’t need to know what causes people who buy a Paleo diet book to also buy beef jerky.

The passive observation of data has limited value, because, as Judea Pearl reminds readers several times in The Book of Why: The New Science of Cause and Effect, data is profoundly dumb. “Data can tell you that the people who took a medicine recovered faster than those who did not take it, but they can’t tell you why,” writes the director of UCLA’s Cognitive Systems Lab. “Maybe those who took the medicine did so because they could afford it and would have recovered just as fast without it.”

Association, which Pearl, a Turing Award winner, identifies as the first of three steps on his ladder of causation, won’t help executives answer many of the questions they need to ask when formulating corporate strategy, making investment decisions, or setting prices. To answer questions such as, “What will raising prices by 10 percent do to revenues?” you need to start climbing Pearl’s ladder. Read the rest here.

Why So Many Data Science Projects Fail to Deliver

Learned a lot lending an editorial hand here:

MIT Sloan Management Review, March 2, 2021

by Mayur P. Joshi, Ning Su, Robert D. Austin, and Anand K. Sundaram

Image courtesy of Jean Francois Podevin/theispot.com

More and more companies are embracing data science as a function and a capability. But many of them have not been able to consistently derive business value from their investments in big data, artificial intelligence, and machine learning. Moreover, evidence suggests that the gap is widening between organizations successfully gaining value from data science and those struggling to do so.

To better understand the mistakes that companies make when implementing profitable data science projects, and discover how to avoid them, we conducted in-depth studies of the data science activities in three of India’s top 10 private-sector banks with well-established analytics departments. We identified five common mistakes, as exemplified by the following cases we encountered, and below we suggest corresponding solutions to address them. Read the rest here...

Want to Make Better Decisions? Start Experimenting

Learned a lot lending an editorial hand here:

MIT Sloan Management Review, Summer, 2020

by Michael Luca and Max H. Bazerman



Image courtesy of Ken Orvidas/theispot.com

Suppose you work on Google’s advertising team and need to decide whether ads should have a blue background or a yellow background. You think that yellow would attract the most clicks; your colleague thinks that blue is better. How do you make the decision?

In Google’s early days, the two of you might have debated the issue until someone caved or you both agreed to kick the decision up to the boss. But ultimately, it dawned on leaders throughout Google that many of these debates and decisions were unnecessary.

“We don’t want high-level executives discussing whether a blue background or a yellow background will lead to more ad clicks,” Hal Varian, Google’s chief economist, told us. “Why debate this point, since we can simply run an experiment to find out?”

Varian worked with the team that developed Google’s systematic approach to experimentation. The company now runs experiments at an extraordinary scale — more than 10,000 per year. The results of these experiments inform managerial decisions in a variety of contexts, ranging from advertising sales to search engine parameters.

More broadly, an experimental mindset has permeated much of the tech sector and is spreading beyond that. These days, most major tech companies, such as Amazon, Facebook, Uber, and Yelp, wouldn’t make an important change to its platforms without running experiments to understand how it might influence user behavior. Some traditional businesses, such as Campbell Soup Co., have been dipping their toes into experiments for decades. And many more are ramping up their efforts in experimentation as they undergo digital transformations. In a dramatic departure from its historic role as an esoteric tool for academic research, the randomized controlled experiment has gone mainstream. Startups, international conglomerates, and government agencies alike have a new tool to test ideas and understand the impact of the products and services they are providing. Read the rest here.

Making experiments pay

strategy+business, June 2, 2020

by Theodore Kinni




Illustration by AndSim

The current pandemic is a dramatic case study in why experiments matter. How else are we to know which tests accurately identify COVID-19 infections and their telltale antibodies, whether and how well drugs already at our disposal mitigate the suffering inflicted by the coronavirus, or whether the vaccines currently in development will actually protect us from it? Randomized controlled trials are the only way to answer these questions short of playing Russian roulette with the lives of large numbers of people.

Experiments have been around for a long time. In The Power of Experiments, an introductory paean to the benefits of experimentation for corporate and government decision making, Harvard Business School professors Michael Luca and Max Bazerman peg the first recorded experiment to the reign of King Nebuchadnezzar in Babylonia circa 600 BC.

As the Old Testament story goes, Nebuchadnezzar attacked Jerusalem and took a group of young Israelites, including one named Daniel, as servants. He ordered that they be acculturated for three years, in part by being fed the same food as the royal court, which the Israelites considered “ritually unclean.” But Daniel proposed that he and three other prisoners be fed only vegetables and water for 10 days, at which point a guard would compare their health to that of the rest of the prisoners, who would have been fed the local fare. At the end of the experiment, the guard judged the vegetarians healthier, and Daniel and his three comrades were able to continue eating the less objectionable diet.

About 2,600 years later, Luca and Bazerman write, “We are in the early days of the age of experiments.” By this, they mean that experiments have spread far beyond the boundaries of science labs and medical trials. In 2018, Google ran more than 10,000 experiments. Amazon, Facebook, Uber, Yelp, and TripAdvisor run thousands of experiments per year. As these names indicate, technology companies, especially those with digital platforms, are at the forefront of corporate experimentation. That’s because it is relatively easy to run experiments on their large, captive audiences. Read the rest here.