Blog: Barry Devlin Subscribe to this blog's RSS feed!

Barry Devlin

As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein.

About the author >

Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

Enterprise BI shops and data quality departments regard spreadsheets largely as the work of the devil. Against all the rules of information quality, data in spreadsheets is manipulated by users at will and in private. Then the resulting data and function is distributed, shared and further played around with, until it's anybody's guess whether the results presented at the end bear any relationship to the truth. Data that was pure and clean as it came out of the data warehouse, data mart or approved BI report is now potentially as contaminated as nuclear waste.

And yet, check in with the users. Indeed, check in with yourself. Why is Excel so popular? Because it makes it easy to play with the data, check out hypotheses, get answers otherwise unavailable, and so on. And once you've gotten the answer through the spreadsheet, chances are you won't get the time or the resources to recreate the process in a more auditable, quality-conscious way. It's a real and spreading problem. But, what to do?

This week I had the opportunity to preview a new product called Lyza that's due to launch on Sept. 22. In fact, you can download it and play with it already. Scott Davis, the CEO of Lyzasoft Inc. explained that they had spent a lot of time investigating how business analysts, the power users of spreadsheets, actually work. This is usually a good idea, because you find out what the users really need, and which of your assumptions are right or wrong. It will probably come as no surprise that most analysts approach their work in a highly unstructured and iterative way, pulling bits of relevant data into Excel from a variety of known sources - both official data marts and reports as well as unofficial files, spreadsheets, etc. they happen to have created before or borrowed from trusted colleagues. And they do it in Excel, because that's the only way they can.

What Lyza does is to provide an easier, more intuitive way of pulling data together from diverse sources, combining and manipulating it and creating results and reports for distribution to the business. Well, that's all fine and dandy for the business analysts you may say, but how does it help the BI and data quality departments address the data contagion? The answer is that Lyza tracks and saves an audit trail of every action and every step of the analysis process that the user is building as well as enabling snapshots of the results to be cached and preserved for posterity. Now the data quality folks are beginning to smile. And the BI department? Well, they're less sure: they like the added traceability, but this is still outside their comfortable data mart zone.

However, we could look at it in a different way. We could imagine that Lyza provides a new type of data mart - a "playmart" - a sand box where power analysts can experiment with data and perform all sorts of analyses in a safe, well-managed environment. Now, if only we could evaluate the analysts' logic and productionalize those analyses and reports that are going to be reused and built upon in the future.

Scott's initial answer was that you can certainly do all this within Lyza itself. But a bit of further probing convinced me that the metadata that Lyza stores to describe the analysis processes is probably sufficient to enable the creation of ETL scripts for your ETL engine of choice. This would certainly require further investigation and automation, but it seems like the bones of the idea are there. In this case, the playmart could address a set of business analysts' needs that have been long ignored by the BI departments and by BI vendors as well.

The only real fly in the ointment is whether Lyza will be able to convince the spreadsheet jockeys to get off their current Excel rocking horse and jump on the bright new Lyza pony in the playmart! (And that sentence would work so much better if only Lyza had chosen a mustang for their logo rather than a gecko.)


Posted September 19, 2008 4:54 PM
Permalink | No Comments |

Leave a comment