Blog: Richard Hackathorn

« Becoming a Real-Time Enterprise: Is the Book worth It? | Main | Mixing Old Wine in a New Wine Skin »

The Black Swan of Business Intelligence

Sometimes you start reading a book with low expectations about its significance. But, the book surprises you and delivers a message of great significance. That has happened with a new book entitled The Black Swan: The Impact of the Highly Improbable by Nassim Nicholas Taleb. He is a professor of the Sciences of Uncertainty (an odd title) at the University of Massachusetts. See his Wikipedia entry and a PBS podcast.

Let me start with the bottom line. I strongly recommend this book for all professionals in Business Intelligence (BI) who care about the means and results of our profession upon our clients.

I have this naïve belief that more information is better, assuming that the information is relevant to the business, properly cleansed, structured cross-functional, analyze appropriately, distributed to the right people and so on. This book totally negated that belief, instilling a humble attitude toward how much we can not know and shocking me about how much our current BI practices do damage to our clients.

And... I have just read the first few chapters. I am starting to be aware of the problems in general, confused about their implications to BI, and wondering whether there are any solutions. This is a book that will take several months to consume (because you read a few sentences, think ‘what?’ and then reread it several more times).

Let me give a small taste of Taleb’s argument. Before Australia was discovered, everyone knew that all swans were white, because all swans that were ever observed were white. Therefore, rule of nature was that all swans are white. Someone discovered a black swan in Australia. That one swan negated a belief held for a thousand years by all of mankind. Afterward, people concocted explanations as to why such a rare animal was perfectly normal and should have been expected. Taleb then extends this analogy to explain the events and aftermath of September 11, along with many other pivotal events in human history.

That is the Black Swan. It is a totally unexpected, but significant, rare event that seems plausible...afterwards. In Taleb’s words, the Black Swan is an event with three attributes: “First, it is an outlier as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact [changing our basic paradigms that explain the world]. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”

I submit that we are unprepared to handle the Black Swan with current BI technology and practices. In fact, current BI does more harm than good, by giving us a false sense of reliability in what we think we know.

Help me with my struggle to understand the practical importance of the Black Swan. I would like to get a discussion established on Black Swan issues within the BI profession, along with joint publications with some of you. Is there anyone interested in this pilgrimage?

Share: