Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

Warning: this is more of a rant, which I usually don't do - my apologies... This came across my desk this morning: IBM announces capabilities to support Dynamic Data Warehousing... It's always interesting when big companies don't give credit where credit is due. Of course, I'm not the first one to discuss DDW, and I certainly won't be the last. On the other hand, IBM is making claims that they came up with the definition... We'll I'll be! I had no idea that anyone was even talking about it. (See my post from 2005, here).

Bill Inmon, myself, Claudia Imhoff, and a few other peers on the COBICC board (Colorado Business Intelligence Community Connection) have been discussing these notions for years. Now don't get me wrong, I applaud IBM for beginning the effort to meet the needs of DDW, but I don't think they should be claiming credit for being the first to create a solution. Ralph Kimball has also been discussing DDW (in a different light)... It's the natural evolution of the warehousing industry.

Here's a quote directly from the IBM marketing brochure:

Dynamic warehousing is not a product, tool or simple one-off solution. It is an approach that enables you to deliver more dynamic business insights by integrating,
transforming, harvesting and analyzing insights from structured and unstructured information. The result is a framework for delivering right-time, contextual information for both strategic planning and operational purposes. Enabling dynamic warehousing requires a set of services that extends beyond traditional data warehousing and reporting to support the increasing number of business processes and applications requiring analytic capabilities, and to address the demands for more dynamic business insight

Interesting, they say it's not a product, tool or one off solution but rather an approach. Then they go on to state that IBM DB2 UDB is the only database that supports this approach, and that you have to hire their consultants to build it.

When will vendors "get it"? And begin to credit the resources that create the terminology and define the industry? You know, my credit goes back to all my mentors - for creating an industry I earn a living in. I can only hope that one day I'm as smart as they are.

They are right about one thing: DDW is the future, but they fail to include neural networking, AI, discovery, semantic ontologies, and dynamic integration. In my mind, they are using the term loosely to define their next level of "EDW appliance". This is what lead my friend Bill Inmon to define and trademark DW2.0, lack of common definition for the term Data Warehouse.

Here's a quote about their new "DDW:"

Traditional data warehousing solutions have provided insurance companies with a valuable tool for analyzing paid claims for possible fraud. Unfortunately, companies often have difficulty recovering funds that have already been dispersed. Dynamic warehousing provides a way to transform this process by aggregating relevant information from across the organization. For example, it can include details that are potentially relevant to claims flight, and embed scoring and analytics directly into the claims review process to identify potential
fraud prior to approval and payment.

Ok, if a Data Warehouse is built properly to begin with, it would have already put together levels of aggregation horizontally across the organization. Furthermore, it would already have had data mining capabilities built in to identify fraud. There is NOTHING dynamic about this definition, where's the automated adaptability to services? Where's the automated data model recognition changes? Where's the automated adaptability to BI queries? Where's the automated discovery of new BI queries and reports?

Here we go again:
Customer service

Most companies have a wealth of information about their customers stored in various systems across the organization but have focused their warehousing efforts on more traditional reporting of customer problems. However, if call center agents can combine the information they have collected about a customer with historical information in the data warehouse—and leverage dynamic analysis capabilities—they can identify other potential issues the customer may be facing. Armed with this knowledge, agents can better recognize the likelihood of a customer leaving while they are talking to the client, improving the chances that they can take steps to maintain the customer’s loyalty. They can even more easily identify appropriate cross-sell opportunities to turn customer support efforts into revenue-generating opportunities.

This is Real-Time Data Warehousing, Active Data Warehousing with a live neural-network on top. This is what banks, credit card companies, and airlines do today... Where's the DYNAMIC part of this warehouse? If the warehouse was built properly to begin with, it would scale, it would run in real-time, and it would have analytics (i.e.: live mining) built in.

Appliances, Teradata, Oracle, HP, Microsoft (SQLServer), and so on are all capable of Real-Time and Active Data Warehousing. Each of the cases listed in this marketing brochure reflect ADW, not DDW. There is nothing said in here that indicates that the new IBM "appliance" with DB2 UDB is dynamic in any way.

ICING ON THE CAKE:

The hub of dynamic warehousing is still the data warehouse. Moving forward, however, data warehouses will need to support increasingly mixed application requirements, including mission-critical operational activities in addition to traditional back-office reporting. Warehouses will also need to provide expanded functionality and work seamlessly with the other services required to enable dynamic warehousing. A data warehouse solution should be able to consolidate data marts and silo solutions across the enterprise into a unified warehouse where needed, while still enabling distributed data marts to address business- specific requirements. And as the demand for resources increases, companies will require more balanced and optimized performance from warehouses (balanced storage, hardware and software performance) to keep costs in check and meet varying service level requirements.

When will these big companies get it? Dynamic Data Warehousing IS NOT YET HERE! IBM Starts by saying that DDW is not a tool, a product, etc... but then at the end defines how IBM DB2 Balanced Warehouse is a DDW... Where's the definition of what DDW really is? In other words, where do they state what part is Dynamic about their system?

The sad part of all this is: IBM has a really great approach, it's just misnamed. It is not DDW, but ADW, it is a great approach to a next generation integrated appliance (with all the software and hardware bundled to make it go). I like the approach, but PLEASE don't call it dynamic data warehousing.

In my next entry I'll re-define Dynamic Data Warehousing and it's evolving state, IBM - you cannot take credit for creating this term (neither can I), what you are defining is Active Data Warehousing, or Real-Time Data Warehousing, not Dynamic Data Warehousing...


Posted June 5, 2007 4:41 AM
Permalink | 2 Comments |

2 Comments

Our industry is hopeless at naming and defining both software categories and products. We all debate until we're blue in the face about what a particular moniker means, and we don't seem to get much closer to a universal consensus than when we had started.

That being said, here's my contribution to the fray. A Dynamic Data Warehouse should be called such only when it can accommodate a number of changes in a development environment and made available for a business user to test quickly enough to be useful to a changing business. For example - adding new reports, fact sources of differing granularities, new ragged hierarchies to existing dimensions, new calendars, etc. should take hours if not minutes. Once we get to this level of change management performance we have a DDW. Poorer performance than this and the business user is left waiting on the IT group yet again for information they need to do their job.

In the course of his polemic, Dan is kind enough to say that I am a friend - I hope that is still the case when he is done reading my reply. He adds that I have provided useful background on dynamic data warehousing in a related publication. Thus, my esteem for his judgment is high, his rant is an interesting one, and it deserves a considered reply. Please find my reply at DMReview.com on-line (6/21/07) --
http://www.dmreview.com/article_sub.cfm?articleID=1086835 "Dynamic Data Warehousing Reaches Take Off Speed"

Leave a comment

    
Search this blog
Categories ›
Archives ›
Recent Entries ›