Blog: Dan E. Linstedt

« Dynamic Data Warehousing and IBM (a rant) | Main | DDW - Detecting and Securing Changes »

Dynamic Data Warehousing... A new definition

I've been writing (scantily) about DDW in the past, in this entry we will take a look at what the definition appears to be in the industry, and then I will offer my opinion on what I think the definition _should_ be for DDW. If vendors believe that they have a DDW, or a DDW solution, then I open heartedly invite them to contact the COBICC board members, and give us all a demonstration, along with definitions of what they've produced.

Dynamic Data warehousing, what does it mean to you?
Throughout the industry we've been getting up to speed on Active or Near-Real Time Warehousing lately, and recently we've also begun experimenting with getting to the next level: DW2.0 (which includes an ADW, structured and unstructured information, metadata, and so on). So what are researchers and folks in the industry saying DDW is?

The first link is to a student’s research project regarding what their view of DDW is:
http://64.233.167.104/search?q=cache:KGeKi0tFzFQJ:www.dblab.ntua.gr/~dwq/p44.pdf+dynamic+data+warehouse&hl=en&ct=clnk&cd=1&gl=us

DWs are dynamic entities that evolve continuously over time. As time passes, new queries need to be answered by them. Some of the new queries can be answered by the views already materialized in the DW. Other new queries, in order to be answered by the DW, necessitate the materialization of new views. In any case, in order for a query to be answerable by the DW, there must exist a complete rewriting 5 of it over the old and new materialized views.

Ok, this is an interesting look - but certainly not the complete picture of what I see DDW to be. To give them credit, they are attacking a difficult problem: how to answer a new query that doesn't have the appropriate data set available - by building new materialized views. The concept is decent, but the words "materialized view" make the approach locked in to Oracle, as other databases do not have the notion of a materialized view. They go on to discuss how to create new views that are needed, and they do a good job of expressing the mathematics behind the desire. Again, this is only one piece of Dynamic Data Warehousing.

Here's another project:
http://xml.coverpages.org/xyleme.html
While they discuss some notions of dynamic data warehousing, they do not disclose all the pieces they will manage. They seem more interested in the fact that they can store vast quantities of XML, and rely on the notions that XML query can change with the XML document structure changing, true - but this still doesn't answer the questions about dynamic restructuring, dynamic indexing (changing indexes when a new one is needed), dynamic query building, dynamic security, and so on. I'll provide this list a little later. However, they are closer to a holistic solution than the first reference.

Here's another interesting look, they start out sounding very promising, but when it comes to brass tacks they are merely discussing Dynamic View Generation - still a worthy cause, but not quite a DDW (as they originally claim).
http://davis.wpi.edu/dsrg/EVE/idm2002-eve.html

IBM has been at it a while, and in this definition - they are defining (you guessed it) an appliance, with bundled software, but in their press release I blogged on yesterday they said DDW is not a tool, a product, or a service... yet again they contradict themselves. Besides that - what they really have is an ADW, not a DDW.... read on... http://www.intelligententerprise.com/showArticle.jhtml;jsessionid=MHPVP5URAXGSAQSNDLRSKH0CJUNN2JVN?articleID=198000675

And another post by Doug Henschen agrees with me.

My friend Lou Agosta does a decent job of discussing some of the background pieces involved in Dynamic Data Warehousing. I think what's missing here is the definition of what Dynamic really means... Should Dynamic mean the data warehouse is dynamic with near-real time data? should it mean it is dynamic with query changes? Dynamic with unstructured data? what does it mean?

Here's another vendor (Axiom Software Labs) that claim to have a DDW, they are probably closer to the mark, but again all of these solutions say they have dynamic abilities, but none of them talk about HOW these abilities work, nor do they disclose what true DDW needs to be. Oh yes, a new acronym is emerging (unfortunately) DyDa - what?

Ok, here are my thoughts on what is required in order to be "the next level" or to be a DDW. We require that all of the following be recognized as dynamic:
* Structural changes to structured data sets are recognized, and changed as available - automated back-room basis.
* Views are adapted as needed when structures change
* Active and Batch loaded data is occurring on the same system at the same time
* Procedural Load routines are adapted to the structure changes when they occur
* Data Mining occurs to build new models against the data in a dynamic fashion
* Architecture mining occurs to determine if the structural changes are attached in the right place.
* Unstructured Data is attached, and searched - all data which can be inserted into a structural matrix will be.
* BI Reports and dashboards are dynamically altered to include the new elements.
* Web services are versioned and re-released to include the new elements.

And so on. Dynamic is a very versatile word, and DDW (in my mind) encompasses a whole lot more than just one piece of the pie (Dynamic Data or Dynamic Views). While these are noble efforts and steps in the right direction, they are _not_ qualified to be called a DDW environment, because they are only pieces of a larger puzzle.

I welcome your comments as always, do you have a definition of DDW that you can share? What is it in your mind?

Thank-you,
Daniel Linstedt
http://www.COBICC.org/

Share: