Now, I may be accused of getting up on my soap box in this first post of 2012, but... a few recent articles on the topic of big data / predictive analytics have really got me thinking.  Well, worrying, to be more precise.  My worry is that there seems to be a growing belief in the somehow magical properties of big data and a corresponding deification of those on the leading edge of working with big data and predictive analytics.  What's going on?

The first article I came across was "So, What's Your Algorithm?" by Dennis Berman in the Wall Street Journal.  He wrote on January 4th, "We are ruined by our own biases. When making decisions, we see what we want, ignore probabilities, and minimize risks that uproot our hopes.  What's worse, 'we are often confident even when we are wrong,' writes Daniel Kahneman, in his masterful new book on psychology and economics called 'Thinking, Fast and Slow.'  An objective observer, he writes, 'is more likely to detect our errors than we are.'"

I've read no more than the first couple of chapters of Kahneman's book (courtesy of Amazon Kindle samples), so I don't know what he concludes as a solution to the problem posed above--that we are deceived by our own inner brain processes.  However, my intuitive reaction to Berman's solution was visceral: how can he possibly suggest that the objective observer advocated by Kahneman could be provided by analytics over big data sets?  In truth, the error Berman makes is blatantly obvious in the title of the article... it always is somebody's algorithm.

It's that time of year when every analyst worth his or her salt is making predictions for the coming year.  Acquisitions.  Big data.  Mobile BI.  Cloud.  Social media.  Predictive analytics... hey! Wait a minute!

My question is: how many of these predictions about BI 2012 are based on the use of predictive analytics?  My hunch is... none.  Perhaps I'm being unfair?  Is it predictive analytics to use all those surveys of buying intentions as input?  What about using trend numbers for market share over the past few years? 

So, here is the Wikipedia definition: "Predictive analytics is an area of statistical analysis that deals with extracting information from data and using it to predict future trends and behavior patterns. The core of predictive analytics relies on capturing relationships between explanatory variables and the predicted variables from past occurrences, and exploiting it to predict future outcomes."  What do you think?  How is the fit?

I wrote a White Paper for an interesting, UK-based start-up, NeutrinoBI, back in October on the topic of freeform search in BI.  So, a new paper by Marti A. Hearst, "'Natural' Search User Interfaces", in the November issue of "Communications of the ACM" caught my attention.  I was particularly interested because Hearst has been one of the main proponents of faceted search, an approach that is relatively unsuccessful in BI.  I wondered if I had missed some new developments in the field.

Sometimes, business intelligence stories can be complex and technical.  Sometimes, even the success stories of a million shaved off expenses here or a 3% improvement in profit margin there can be a bit repetitive.  However, it is the success stories that can prove the business value and quality that characterize the best BI projects.  As one of the judges of the annual South African BI Excellence and Innovation Awards (closing date for entries is 30 November), I'd like to share with you a novel approach to making your entry stand out.  For those of you not eligible to enter, the approach also works when trying to raise funding from the business for a new BI project.  However, you may need to be very brave, even foolhardy; sometimes, you need to tell the as-is horror story to prove just how much better the to-be situation will be.

This story that I'm about to summarize comes courtesy of Teradata's Bill Franks, who posted it about 10 days ago on the Smart Data Collective blog, where it's attracting lots of attention.  Entitled "The Dire Consequences of Analytics Gone Wrong: Ruining Kids' Futures", Bill recounts the story of a local school who invested for the first time in an analytics package designed (allegedly) to detect cheating in English essays.  The package was run for the first time against a set of essays submitted at the start of term by a class of highly motivated and high performing students.  The package promptly reported that each and every student was a cheater, with pervasive copying and plagiarism throughout the group.  The school failed all of the students on the assignment and was about to note the offense on the students' records, an action that would have had severe consequences for their future educational chances, until the parents stepped in...

I leave you to check the rest of the story on Bill's post.  But the bottom line was that the school and the teachers trusted the results of the package more than their own prior experience of the students.  A result of stunning implausibility from the software was accepted without question, without any resort to reason or common sense.  Apparently, the school backed down on noting the offense on the students' records; but it stood by the decision to fail every student in the class on the essay.

As BI practitioners, I trust you get the message.  Analytics in the hands of naive users can be more dangerous than a Kalashnikov.  Self-service BI may speed up delivery, but how reliable are the conclusions?  There's a good reason why assault rifles are not available on open supermarket shelves (in most countries!).  I'm not against self-service BI, but I do have a problem with willful and uneducated business users.  BI must complement common sense, not to substitute for it.  It's up to IT to ensure the users are informed of the strengths and weaknesses of new tooling.

To return to your entries for the BI Awards, to be presented at the BI Summit in Johannesburg on 28 February next, I hope you don't have a horror story of such magnitude, and if you did, I imagine your CEO would be reluctant to have it told in public.  But do remember that the difference between the before and after pictures is a strong indication to the judges of the value of the project to the business.  BI Excellence, beyond technical architecture, also includes data governance and user education.  BI Innovation is about changing the way users make business decisions.  Good decisions based on reliable information.

"Seven Faces of Data - Rethinking data's basic characteristics" - new White Paper by Dr. Barry Devlin.

We live in a time when data volumes are growing faster than Moore's Law and the variety of structures and sources has expanded far beyond those that IT has experience of managing.  It is simultaneously an era when our businesses and our daily lives have become intimately dependent on such data being trustworthy, consistent, timely and correct.  And yet, our thinking about and tools for managing data quality in the broadest sense of the word remain rooted in a traditional understanding of what data is and how it works.  It is surely time for some new thinking.

A fascinating discussion with Dan Graham of Teradata over a couple of beers in February last at Strata in Santa Clara ended up in a picture of something called a "Data Equalizer" drawn on a napkin.  As often happens after a few beers, one thing led to another...