We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

I can't decide if this fits under nanotech or if it fits here, but I'll put it in this category, and focus on the business sides of the house. The winds are blowing outside today, as I sit here anxious for a return call. I've contacted a few individuals at a university which is currently studying structural mining techniques, and will hopefully be discussing some of their progress soon.

In this entry, we will explore the brave new world of what I like to call: Dynamic Data Warehousing. I'm not referring to Dynamic Data sets, I'm referring to Dynamic Structuring and Restructuring of the information systems as a whole.

What is Dynamic Data Warehousing?
I am defining the term as follows:
The ability of a system to 1) interrogate arriving information at run-time 2) discern new "structure" from old "structure" 3) separate the new structure, and build or attach new structural elements to the existing structure, 4) mine existing structural elements for unseen relationships and finally 5) Follow a series of "alert" patterns to notify operators that new nodes or elements have been added, and need to be checked.

In a business sense, or the simplistic definition is: to add and/or change the structure of information on the fly based on "content analysis". The adaptation of the structure is in near-real time, and will result in learning things we didn't know before. It basically changes the data model underneath the covers by using neural net techniques and structural analysis ideas.

Why would I want Dynamic Data Warehousing?
Well, for one - it's convergence (see my nanotech articles here for the series on convergence) of both form and function. Why do we want to converge the two? The electronic computing world is already far behind other sciences and advancements, it's time to UPDATE. It almost feels like we're stuck in the '70s. Ok, here's a reference to Bio-Informatics that talks about what nature does with DNA and convergence of form and function: IEEE Magazine.

What is structural mining and why would you want it?
We've been mining our data sets for years, why not our architecture? What kind of insights would we find from profiling and mining the architecture itself? We might find holes in the source system processing, we might find better methods to re-organize the data underneath (make more sense out of it), we might find relationships between structures that "today" don't have any built.

Structural Mining, or structural analysis is the ability to find out what's right and wrong with the architecture. The ability to discover new and different methods for storing, retrieving and hooking data up. Structural mining is a key component of Dynamic Data Warehousing (could be Dynamic Data Integration too), and the ability to change structure on the fly.

Imagine this: you build a web-service to accept incoming transactions from a provider. Today, it has name and address on it. Tomorrow you ink a deal for them to provide city, state, and zip. It shows up on the feed that night. Let's say that IT "hasn't gotten around to changing the structure" yet, and you have structural analysis engine applied to the service. No sweat, the new fields arrive, and they are in context of the customer record - the SAE (structural analysis engine) doesn't see any harm in automatically adding the fields to the data model, and proceeding with the load.

This is a level 3 change (scale of 1 to 3, 1 being Manual intervention needed before change, 2 being warning: change occurred - 60% to 80% sure that it works, 3 being no problem, context determined with 90% or better confidence rating - change applied).

From time to time (as with all neural nets) we'd have to correct the neural model that the SAE has built, but for the time being, it becomes a central part of the glue to building a Dynamic Data Warehouse (or Dynamic Data Integration store).

On the flip side, it would mean learning some lessons about Fraud detection, and teaching those to the SAE as well - so that it can spot potentially fraudulently added data trying to get in to the system. A gate-keeper of sorts.

I believe that Dynamic Data Warehousing or Dynamic Data Information Stores are the next level of integration, however to get there - it requires a data modeling technique that is capable of being altered without losing existing information or corrupting existing structural integrity.

What might be the ROI on something like this?
Well, that's anyones guess. But I would gather a hunch that if "cleaning up the data sets" can garner 200% ROI or more, then cleaning up the architecture it lives in could be a 4x to 10x multiplier (pure speculation on my part).

Thoughts? Would love to hear your comments on this.

References:
Enterprise GIS Architecture, DDW
Dynamic View Alteration
Comments on Axiom Software DDW
Cross-Linkages with quite a few White Papers listed
Data Warehouse Configuration
Real Time Road Mapping
Percipio Tool for Dynamic Data Warehousing
ENTER GOOGLE SEARCH TERM: "Dynamic Data Warehouse"


Posted May 6, 2005 2:10 PM
Permalink | 3 Comments |

3 Comments

Is dynamic data warehouse a data quality based concept?

Interesting question, let me address this in another blog entry.

I was surfing the net and saw this post from last year. I recently joined iStrategy to assist in their launch of a Data Warehouse solution focused on Higher Education. Interestingly, most institutions are swimming in an ocean of data but can't get the information they need out of the ERP systems to turn it into real knowledge.

Our solution utilizes a dynamic reporting model to enable colleges and universities to unleash the information that is buried in their transactional systems. This leads to better decision-making, improved student retention and recruitment and the distribution of knowledge outside of the report writers.

I'd be interested to hear more about your research on data warehousing.

Leave a comment

    
Search this blog
Categories ›
Archives ›
Recent Entries ›