We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Blog: Neil Raden Subscribe to this blog's RSS feed!

Neil Raden

I hope that you will engage with me with your comments as we explore the future of business intelligence (BI), particularly its expanding role in the actual process of making decisions and running an organization. BI is poised for a great leap forward, but that will leave a lot of people and solutions behind so expect a bumpy ride. I also expect there will be a flurry of advice and methodologies for moving BI into a more active role, one that will widen the audience as BI meets more needs. But a lot of that advice will be thin and gratuitous, so hold on while we put it under the microscope. You can reach me directly if you prefer at nraden@hiredbrains.com.

About the author >

Neil Raden is an "industry influencer" – followed by technology providers, consultants and even industry analysts. His skill at devising information assets and decision services from mountains of data is the result of thirty years of intensive work. He is the founder of Hired Brains, a provider of consulting and implementation services in business intelligence and analytics to many Global 2000 companies. He began his career as a casualty actuary with AIG in New York before moving into predictive modeling services, software engineering and consulting, with experience in delivering environments for decision making in fields as diverse as health care to nuclear waste management to cosmetics marketing and many others in between. He is the co-author of the book Smart (Enough) Systems and is widely published in magazines and online media. He can be reached at nraden@hiredbrains.com.

Bill Inmon posted an article  where he discussed the drawbacks of a "virtual data warehosue." Now the idea of a virtual data warehouse has been around for years (I remember IBI positioning their EDASQL technology as a virtual data warehouse fifteen years ago). That was a terrible idea then because it provided only a method to access data, with no proper cleansing, integration or persistent storage of it. It was, frankly, not a data warehouse at all. Using that definition, anyone would agree that a "virtual data warehouse" was not a workable solution.

The problem is that Inmon defines any method to augment the data warehouse with access to source data directly as a virtual data warehouse, and in the same terms as the IBI concept from long ago, citing its shortcomings, casting the virtual data warehouse as not only a bad idea, but one that has been historically bad and discredited. In fact, the term "virtual data warehouse" has been used for years by detractors for any idea that augmented or supplemented the data warehouse architecture, many of which were truly misguided ideas. In today's world, though, it is not only possible, it is necessary to supplement the data warehouse. This "virtual data warehouse" is a new thing, something not possible before.

Inmon's primary argument about virtual data warehousing is that it skips the crucial step of integrating data:

The great appeal of the virtual data warehouse is that you do not have to face the problem of integration. 

This is the central premise of his argument, that a virtual data warehouse skips data integration.  What actually happens in today's version of a virtual data warehouse (and it shouldn't really be called that) is that a central data warehouse endures, with the kind of integrated data that Inmon would approve of. However, since there are so many data sources now, they change more frequently and the volumes are extreme, the batch load data warehouse is burdened with latency and often impractical for 100% of the analytical requirements. Because system architecture has changed so radically over 15 years, it is possible to read logs, queues, perform changed data capture and even go directly into the operational systems without resorting to extraction into staging areas or degrading their performance. The semantics of these source systems are considerably more transparent than they used to be and the speed of processing is so much faster it is possible to cleanse and integrate data on the fly. In those cases where it is not feasible, or where for a myriad of reasons it makes sense to warehouse the data, that process is preserved. So the virtual data warehouse is actually more of a surround strategy than an alternative. 

Why then is the virtual data warehouse such a supremely bad idea? There are actually lots of reasons for the vacuity of virtue manifested by the virtual data warehouse. 

I won't repeat all of his reasons, but they all seem to be related to the old "it will bring the system to its knees" claims about the inefficiency of federated queries and the affect on systems and network performance. I call this reasoning "managing from scarcity," except there is no scarcity anymore. Again, not every query is federated; many will be satisfied by the data warehouse. We can build systems thousands of times larger and millions of times faster than we could two decades ago as data warehousing was catching on and isolation of data and processing was absolutely necessary. Today's applications and business processes call for faster, fresher data than a data warehouse can provide, in many cases. Rather than ignore these requirements, a virtual data warehouse (with adequate semantic rationalization) gives the developers a chance to position the processing to the logical location, and the ability to use abstraction (or indirection if you prefer that word) to address the data, providing the flexibility to change that configuration whenever necessary. 

The ugly truth is that the corporate analyst MUST do integration whether he or she wants to or not. There is no getting around it. Wishful thinking just does not cut it here.

Who wouldn't agree with that? The question is, where and when? The bulk of the difficult integration work is manual - people gathering information and getting buy-in that they have the right mapping. Where that process gets implemented is not the issue. There is no rule that says it must all go to a data warehouse. Historically, that was the practice because of the constraint of resource limitations that are now mostly relaxed.

So the problems with the virtual data warehouse are legion. There is poor performance or no performance at all. There is an enormous system overhead. There is only a limited amount of historical data. There is the work of integration that each analyst must do, one way or the other. There is no reconcilability of data. There is no single version of the truth for the corporation.

All of these things are true in the virtual data warehouse the way Inmon defines it, a big mess of federated queries directly to source systems with no integration, no history, no real context. No one would suggest this is a good idea. But a data warehouse today should be a participant in a constellation of source data, so long as the access methods to data beyond the warehouse's control can be relied upon to be correct, timely and not a burden on other systems. This kind of arrangement is not a virtual data warehouse, it's an analytical framework. We should discuss the merits of the idea as such, not dismiss them because we name them with a long-discredited concept. 


Posted March 23, 2009 2:24 PM
Permalink | 5 Comments |

5 Comments

Naturally, we agree with Neil. ("We" are Information Builders; "IBI" is a deprecated abbreviation for "Information Builders, Inc.")

If we stick with the physical warehouse metaphor, some products are in such demand and change features infrequently enough that they go into the warehouse. Custom-built, fast-changing, dynamic things are handled with a mechanism that suits them better. There's nothing inherently wrong with going back to an operational system, especially if it's the system of record for certain information types.

I'm not sure I understand the concern with the corporate analyst doing integration. If it's easy enough to do on the fly, then you can spend the same amount of time on definitions and query-building, but the output will be a report instead of an ETL job. That's more productive, not problematic.

Thanks for your post!

Hi Jake,

Thanks for your comments. I guess it's too easy to fall to abbreviations instead of spelling out names, my apology. I obviously use IBI as shorthand.

Would you do me a favor and take another shot at your second paragraph. I've read it twice and I'm still not sure I understand it.

Good to hear from you.

-NR

Re:IBI, no problem. Lots of us still do it, too. Was meant to be informative, not snarky.

The term "data warehouse" was coined because of a metaphor with a physical warehouse: everything moved to the central location so that it could be appropriately managed and distributed.

Modern logistics doesn't always work that way, though: one-off shipments might go direct from manufacturers, etc. Whatever works best for the business is what's used; the warehouse is important, and most goods are managed there, but we're not ideological about using a warehouse for logistics. Use what works.

Similarly, modern BI doesn't need to all come from a data warehouse. There's nothing wrong with going back to another system for information if that's what works best for the business; the data warehouse is important, and most queries are managed there, but we're not ideological about using a data warehouse for BI. Use what works.

Better?

Neil, what I like most about your post is the balance. The truest truism regarding analytics today is there is no single silver bullet. Whether the bullet is a tool, architecture, report, language, source, etc -- there are simply too many perspectives, too may unique questions, too many new situations, too much fluidity in the environment, too much urgency, etc. to adhere to dogma regarding ANY single approach.

Wise leaders are not dogmatic; they are principle-driven and adaptive. The principle here is competitive advantage in decision-making, and clearly there are trade-offs to be made in terms of quality vs. scope vs. timeliness vs. participation vs. other things. Different situations, even within the same organization, will call for different approaches to generate competitive advantage via decisions/analytics.

Adaptation is more than simply opportunism. It is recognition of shifts that change the possible. When the data warehouse first entered our lexicon, things were a little bit different than today... in what tech can do, as well as in what people can do. But, most importantly, the gentle drizzle of information that was once our operating MO, is now more like being inside a dishwasher -- tons of high-pressure streams, from every conceivable direction. So, there are situations today in which the loss of time-relevancy incurred to pass all useful information through a centrally-controlled/provisioned decision is simply too high a price.

The data warehouse and all its attendant disciplines are not going away. But, it's a wise person who diversifies her arsenal.

Neil and Scott -

I am pleased to see this debate. Vive la difference!

Clearly physical data warehouses will continue at the center of the data integration universe (several big hoses in Scott's dishwasher analogy), but as Scott notes, why limit one's arsenal of tools.

Especially as Neil states, there are any number of complementary virtual federations that are enterprises and govenment agencies are finding useful now that computing resources (as well as sources) are in greater abundance.

-Bob

Leave a comment