We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

I've had several good conversations with different folks in the industry lately about this term and what it means. Lou Agosta was nice enough to write a piece on DMReview, as well as have a really neat phone call with me. We both agree that there are more layers to this onion that originally thought. What a surprise!! But like anything, there are also steps, types, and classes of DDW to eventually get us there. In this entry I dive into the topic of "classifications of terms", and levels of DDW, and attempt to put down a rough road-map to get there. Keep in mind the definition is still in flux, and will be for some time to come.

Dynamic Data Warehousing terminology must be separated into its constituent parts in order to be understood properly (my belief anyhow).

Just like Master Data Management is pulled into: Master Data, and Data Management, and Master Data Management, such is the definition of DDW: Dynamic Data, and Data Warehousing, and Dynamic Data Warehousing

I think each stage of these terms warrants explanation, so here goes:

1. Dynamic Data - something that has been discussed for many many years, and has been affectionately known as Real-Time data, operational data, tactical data, active data, and so on... In other words, it's the data that is dynamic, changing, and responding to the business needs. This particular set of words has zip zero zilch to do with data warehousing, structural integrity, views, queries, ETL, ELT, EAI, and so on - it has everything to do with the data.
2. Data Warehousing - well we all have a pretty good idea as to what this is, and Bill Inmon just solidified the definitions within the architecture of DW2.0. We'll write more on this topic later. Again, how is Data Warehousing dynamic? Well, if you go back to "real-time data" or dynamic data, or active data, you'll see some of the ties. The data warehouse simply becomes a place where both tactical and strategic data can exist. Does this mean that the EDW and the ODS are one in the same? NO - that's NOT what I said, what I said is the "data warehouse" (which includes the ODS (Interactive sector), and a strategic data storage area (Integrated Sector) exist within the same "architectural foundation" called a data warehouse.
3. So where does that leave Dynamic Data Warehousing? Good question, I hope this is one we can answer with many more discussions and entries to come.

Here are my thoughts: When I talk about a "DATA WAREHOUSE" I usually refer to all the moving parts and pieces that make up this "electronic data store", in other words, I include the structure (data model), execution code (SQL queries, loader scripts, unload scripts), data migration code layers (ETL, stored procedures, functions), and of course the metadata (semantic meaning that defines all these things, and hopefully how they all interact).

So when I talk about "DYNAMIC DATA WAREHOUSE", what I'm really saying is:

The following layers must be "included" in the term DYNAMIC:
1. structure and indexing (DDL)
2. integration code
3. SQL Queries
4. BI Reports (some use the queries, some house queries OUTSIDE the RDBMS)
5. Metadata and Semantic Ontologies
5a. This includes: Dependency chains
5b. Workflows (technical workflows)
5b. Data Model Definitions
5c. Aggregate definitions
5d. Security and access concerns .... and so on.

What I mean by Dynamic is the nature of handling change, or as Lou put it: "changing at the speed of business". I would also go so far as to say, "it is AUTOMATED ADAPTATION that enables change at the speed of business."

Now, what does all this mean? To me, Dynamic Data Warehousing is a solution requiring people, machines, standards, processes, architecture, and design elements. In the future once standardization can be executed properly (even within a single DDW within a vendor producing their own "flavor") then it will become possible to AUTOMATE many (not all, but many) of the functions that we currently do by hand.

In other words, like everything else - a "DDW" will follow the commoditization path, the same way appliances are following it. In fact, I'd also be so bold as to say I think Appliances are the right foundation to START with, to move in the direction of the purist DDW. I also think that DDW is a GOAL, not necessarily a solution - some of the parts to what I've outlined may never come to pass, we'll just have to wait and see.

What I will say is this: I know that semantic integration of structures can be done, I know that matching structures to semantic meanings of well-established ontologies can also be done. I also know through a variety of inference engines, neural net algorithms, and visualization that the strength of these relationships can be SEEN, EDITED, and TAUGHT (to the neural net to process this information better the next time). I've seen these software components, they exist today.

I also know that there is still a long way to go, that the results of these data model integrations must be checked manually and corrected - post generation. Furthermore, the only piece that this addresses is the structural component of the DDW. It does not address the automated adaptation of code layers, SQL layers, ETL layers, security layers, metadata layers, nor reporting layers.... these are all things that must be worked on.

For now, all I'm suggesting is "broaden your minds..." Harry Potter movie, art of divination scene. Let's go for the utopian vision, and make our efforts that much better as we progress. After all - a philosopher once said: you set your own limits, and another said: you can only execute to the goals that you set for yourself.

So why not stretch a little bit? In the next entry on DDW, I'll try to unfold the layers - but first, if I missed any layers, please let me know by responding. Furthermore, MARKETING DEPARTMENTS TAKE NOTICE: PLEASE make sure your wording matches your marketing statements, in my last post I took a shot at IBM marketing, I was _NOT_ taking shots at the DB2 UDB product.

Just don't say things to me like: "XXXXXXX is a SOLUTION, not a product, not a this, not a that.... [then later say] YYYYYY is our product, and it is a XXXXXXXX." This is contradictory.

Dan Linstedt
Come get a masters of science in BI at: http://www.COBICC.org

Posted June 27, 2007 6:27 PM
Permalink | No Comments |

Leave a comment

Search this blog
Categories ›
Archives ›
Recent Entries ›