Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

October 2008 Archives

But very few (if any) actually execute on the vision that I am laying out here. This is a very short entry, but basically re-iterates some of the points of Dynamic Data Warehousing that I believe to be necessary before it (software/appliance/database) can be labled as being something like this.

In my definition of dynamic data warehousing the software around the database is an artificially intelligent engine. The database contains metadata about the structure, about the usage of the structure, and versions of all this metadata (producing a structural and usage life-cycle).

In other words, the AI engine is fed or kick started with an ONTOLOGY. The ontology of terms defines the basic data model that is executed underneath. The Ontology is driven by business terms, business definitions, functions, and descriptions (in accordance with OWL ontology). Secondarily, the AI engine is fed many different data points including usage of the ontology:

* SQL Queries
* Loading Code
* Scripting Code
* Application Code
* Web Service Code

And all of the table references/usages/join criteria components within the code.

Dynamic Data Warehousing RESPONDS to changes BY ITSELF. It responds to USAGE controls (ad-hoc queries, repeated queries, and so on). It responds to LOADING controls (changes to structures, appearance of new attributes/fields, changes to loading code, volume and width). It responds to length of processes (metrics driven), and responds to USER DRIVEN ONTOLOGY CHANGES (based on business requirements).

At the end of the day, the AI engine grades changes, and figures out by itself, how to a) TUNE the structure b) ADAPT or CHANGE the structure, including indexing, c) Add new elements to the structure, d) retire old elements from the structure, d) OPTIMIZE the modeling paradigm for today's business execution cycles, e) manage and propogate structural changes TO the loading code, TO the queries, and TO the ontologies.

Vendors may claim that they have "Dynamic Data Warehouses" all they want, but until they have automatic detection of structural changes, and automatic propogation of those changes - and these automated systems are associated to/with business ontologies, I will not agree that they infact have a "Dynamic Data Warehouse".

This is just my opinion. I believe that in order to become more fluid, and more appropriate to the business, and closer to business change, these are the kinds of systems that will evolve in the next 5 to 7 years.

Cheers for now,
Dan Linstedt
Feel free to contact me directly: DanL@GeneseeAcademy.com, we teach custom Informatica courses, DW2.0 and Unstructured Data Courses, Zachman Framework courses, and Data Vault Data Modeling courses.


Posted October 29, 2008 11:31 PM
Permalink | No Comments |