Blog: Barry Devlin Subscribe to this blog's RSS feed!

Barry Devlin

As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein.

About the author >

Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

A chat with Max Schireson, President of 10gen makers of MongoDB (from humongous database), yesterday provided some food for thought on the topic of our assumptions about the best database choices for different applications.  Such thinking is particularly relevant for BI at the moment, as the database choices expand rapidly.

But first, for traditionalist BI readers, a brief introduction to MongoDB, which is one of the growing horde of so-called NoSQL "databases", some of which have very few of the characteristics of databases.  NoSQL stores come in a half a dozen generic classes, and MongoDB is in the class of "document stores" along with tools such as Apache's CouchDB and Terrastore.  Documents?  If you're confused, you are not alone.  In this context, we don't mean textual documents used by humans, but rather use the word in a programming sense as a collection of data items stored together for convenience and ease of processing, generally without a predefined schema.  From a database point of view, such a document is rather like the pre-First Normal Form set of data fields that users present to you as what they need in their application.  Think, for example, of an order consisting of an order header and multiple line items.  In the relational paradigm, you'll make two (or more) tables and join them via foreign keys. In a document paradigm, you'll keep all the data related to that order in one document.

Two characteristics--the lack of a predefined schema and the absence of joins--are very attractive in certain situations, and these turn out to be key design points for MongoDB. The lack of a schema makes it very easy to add new data fields to an existing database without having to reload the old data; so if you are in an emerging industry or application space, especially where data volumes are large, this is very attractive.  The absence of joins also plays well for large data volumes; if you have to shard your data over multiple servers, joins can be very expensive.  So, MongoDB, like most NoSQL tools, play strongly in the Web space with companies needing fact processing of large volumes of data with emergent processing needs.  Sample customers include Shutterfly, foursquare, Intuit, IGN Entertainment, Craigslist and Disney.  In many cases, the use of the database would be classed as operational by most BI experts.  However, there are some customers using it for real-time analytics, and that leads us to the question of using non-relational databases for BI.

When considering implementing Operational BI solutions, many implementers first think of copying the operational data to an operational data store (ODS), data warehouse or data mart and analysing it there.  They are immediately faced with the problem of how to update the informational environment fast enough to satisfy the timeliness requirement of the users.  As that approaches real-time, traditional ETL tools begin to struggle.  Furthermore, in the case of the data warehouse, the question arises of the level of consistency among these real-time updates and between the updates and the existing content.  The way MongoDB is used points immediately to an alternative, viable approach--go directly against the operational data.

As always, there are pros and cons.  Avoiding storing and maintaining a second copy of large volumes of data is always a good thing.  And if the analysis doesn't require joining with data from another source, using the original source data can be advantageous.  There are always questions about performance impacts on the operational source, and sometimes security implications as well.  However, the main question is around the types of query possible against a NoSQL store in general or a document-oriented database in this case.  It is generally accepted that normalizing data in a relational database leads to a more query-neutral structure, allowing a wider variety of queries to be handled.  On the other hand, as we saw with the emergence of dimensional schemas and now columnar databases, query performance against normalized databases often leaves much to be desired.  In the case of Operational BI, however, most experience indicates that the queries are usually relatively simple, and closely related to the primary access paths used operationally for the data concerned.  The experience with MongoDB bears this out, at least in the initial analyses users have required.

I'd be interested to hear details of readers' experience with analytics use of this and other non-Relational approaches.


Posted June 17, 2011 5:50 AM
Permalink | 1 Comment |

1 Comment

Thanks for this clear explanation of what "documents" are and the pros and cons of a document-oriented database versus a traditional RDBMS. It is the best, most jargon-free explanation I've found (and I've been searching for a bit!)

Leave a comment