Blog: Dan E. Linstedt

« Indexing: VLDW and Data Sets | Main | Operational Data Warehousing / Active Data Warehousing »

Thoughts about Dynamic Data Warehousing & Context

I've been discussing DDW for quite a while; I've started discussing the nature of dynamic structure change. There are larger considerations out there that we need to think about before embarking down these paths. However that said - there are some applications regarding architectural mining and dynamic structure changes which I wish to discuss here. For those of you in the intelligence sectors of government or research and defense this may be of interest (or not). For those of you in DW / BI traditional, the only benefits that dynamic structure change might bring to you is the ability to adapt faster (on the back end) to the dynamic changes of business. But then again, this technology is years away (for all we know ;)

Dynamic re-structuring of structured data, why would we do it? What is the interest? What are the benefits?

Well, if you're in the intelligence sector, or identity analytics, or defense research, then this may hold some serious value - and perhaps, you are already performing these tasks - after all, DARPA began funding Nanotech and DNA computing experiments over 10+ years ago (at least as far as we can tell publicly). Enough said...

Anyhow, imagine a system beyond master data.. Where we have the structures that house specific "images" of data at a specific point in time, then we can stack those images and slice by time, or... slice by association.

What do you mean, slice by association?
What I mean is: imagine for a minute that a slice in time combined with a business key, surrounded by specific descriptors actually establishes a particular context. Now imagine that you have several hundred of these data points (multiple keys across the model), each is already imaged by "time", and quite possibly multiple time frames. Finally imagine that the business keys are essentially useless, except for hard mechanisms for people to indentify the information.

Now you are beginning down the path of something called identity analytics. Surround those keys with the notions of context, of course, using the term losely - in other words it is one view of information at a particular point in time, context is how you "rotate" the information to meet the needs of the current end user.

So you're saying "no relationships"?
yes, that is one of the things I'm saying. Now apply data mining across the information sets, and look for previously unknown patterns, but deeper than that - look for abstracted patterns of correlated data - where outliers aggregate in coincidental time frames - now this is presented to an end-user, in a visual 3 dimensional graph format.

We apply color to "Hot" "cold" and luke-warm correlations, the user applies the human thought process of "interest". By focusing in on the interested points, and applying human logic we could theoretically surf billions of contextual relationships that would otherwise go un-noticed.

Now, the human interaction establishes (interactively) the points of interest or the relationships that are associating the information to other points of information. Once this is done, a new set of data mining algorithms are run. These algorithms produce a specific answer, and test correlation of information to a more focused lens. This cycle can be run over and over again until the human decides that the relationship is of interest, and NOW can apply information relationships dynamically.

Once this relationship "falls out of interest" it is removed, in favor of a new relationship. In essence, the model becomes a slowly evolving model with human intervention. It's possible that after certain relationships have been identified, that the data mining algorithms can be "tuned" to self-modify parts of those relationships.

Well, all of this is just a thought experiment - the only part which may not necessarily be achievable today is the application of these changes to the queries, and load routines. Certainly without human interaction, zooming in to points of interest becomes a difficult task.

Identity analytics plays a role like this, in identifying context from information - then relating different "identities" as associated elements. But that's for another day.

I hope you found this entry interesting; I'd love to hear your thoughts.

Thanks,
Dan Linstedt
DanL@GeneseeAcademy.com

Share: