TechSavvy: Beware the Paradox of Automation

MIT Sloan Management Review, October 20, 2016

by Theodore Kinni
Earlier this year, Facebook exorcised those pesky human editors who were introducing political bias into its Trending news list and left the job to algorithms. Now, reports Caitlin Dewey in The Washington Post, the Trending news isn’t biased, but some of it is fake. Turns out the algorithms can’t tell a real news story from a hoax.

Facebook says it can improve its algorithms, but errors of judgment aren’t the only pitfall in transferring human tasks to machines. There’s also the paradox of automation. “It applies in a wide variety of contexts, from the operators of nuclear power stations to the crews of cruise ships, from the simple fact that we can no longer remember phone numbers because we have them all stored in our mobile phones, to the way we now struggle with mental arithmetic because we are surrounded by electronic calculators,” says Tim Hartford in an excerpt published by The Guardian from his new book, Messy: The Power of Disorder to Transform Our Lives. “The better the automatic systems, the more out-of-practice human operators will be, and the more extreme the situations they will have to face.”

Hartford borrows William Langewiesche’s harrowing description of the crash of Air France Flight 447 to illustrate three problems with automation: “First, automatic systems accommodate incompetence by being easy to operate and by automatically correcting mistakes. … Second, even if operators are expert, automatic systems erode their skills by removing the need for practice. Third, automatic systems tend to fail either in unusual situations or in ways that produce unusual situations, requiring a particularly skillful response.”

The excerpt is worth a read — especially if it prompts you to ask if your company’s automation initiatives might entail similar risks. Read the rest here.