Blog: Dan E. Linstedt

« Big Data = Big Problems = Huge ROI if done right. | Main | Profitability in all business cycles - part 1 »

Structural Mining, Dynamic Data Warehousing, Neural Nets

I can't decide if this fits under nanotech or if it fits here, but I'll put it in this category, and focus on the business sides of the house. The winds are blowing outside today, as I sit here anxious for a return call. I've contacted a few individuals at a university which is currently studying structural mining techniques, and will hopefully be discussing some of their progress soon.

In this entry, we will explore the brave new world of what I like to call: Dynamic Data Warehousing. I'm not referring to Dynamic Data sets, I'm referring to Dynamic Structuring and Restructuring of the information systems as a whole.

What is Dynamic Data Warehousing?
I am defining the term as follows:
The ability of a system to 1) interrogate arriving information at run-time 2) discern new "structure" from old "structure" 3) separate the new structure, and build or attach new structural elements to the existing structure, 4) mine existing structural elements for unseen relationships and finally 5) Follow a series of "alert" patterns to notify operators that new nodes or elements have been added, and need to be checked.

In a business sense, or the simplistic definition is: to add and/or change the structure of information on the fly based on "content analysis". The adaptation of the structure is in near-real time, and will result in learning things we didn't know before. It basically changes the data model underneath the covers by using neural net techniques and structural analysis ideas.

Why would I want Dynamic Data Warehousing?
Well, for one - it's convergence (see my nanotech articles here for the series on convergence) of both form and function. Why do we want to converge the two? The electronic computing world is already far behind other sciences and advancements, it's time to UPDATE. It almost feels like we're stuck in the '70s. Ok, here's a reference to Bio-Informatics that talks about what nature does with DNA and convergence of form and function: IEEE Magazine.

What is structural mining and why would you want it?
We've been mining our data sets for years, why not our architecture? What kind of insights would we find from profiling and mining the architecture itself? We might find holes in the source system processing, we might find better methods to re-organize the data underneath (make more sense out of it), we might find relationships between structures that "today" don't have any built.

Structural Mining, or structural analysis is the ability to find out what's right and wrong with the architecture. The ability to discover new and different methods for storing, retrieving and hooking data up. Structural mining is a key component of Dynamic Data Warehousing (could be Dynamic Data Integration too), and the ability to change structure on the fly.

Imagine this: you build a web-service to accept incoming transactions from a provider. Today, it has name and address on it. Tomorrow you ink a deal for them to provide city, state, and zip. It shows up on the feed that night. Let's say that IT "hasn't gotten around to changing the structure" yet, and you have structural analysis engine applied to the service. No sweat, the new fields arrive, and they are in context of the customer record - the SAE (structural analysis engine) doesn't see any harm in automatically adding the fields to the data model, and proceeding with the load.

This is a level 3 change (scale of 1 to 3, 1 being Manual intervention needed before change, 2 being warning: change occurred - 60% to 80% sure that it works, 3 being no problem, context determined with 90% or better confidence rating - change applied).

From time to time (as with all neural nets) we'd have to correct the neural model that the SAE has built, but for the time being, it becomes a central part of the glue to building a Dynamic Data Warehouse (or Dynamic Data Integration store).

On the flip side, it would mean learning some lessons about Fraud detection, and teaching those to the SAE as well - so that it can spot potentially fraudulently added data trying to get in to the system. A gate-keeper of sorts.

I believe that Dynamic Data Warehousing or Dynamic Data Information Stores are the next level of integration, however to get there - it requires a data modeling technique that is capable of being altered without losing existing information or corrupting existing structural integrity.

What might be the ROI on something like this?
Well, that's anyones guess. But I would gather a hunch that if "cleaning up the data sets" can garner 200% ROI or more, then cleaning up the architecture it lives in could be a 4x to 10x multiplier (pure speculation on my part).

Thoughts? Would love to hear your comments on this.

References:
Enterprise GIS Architecture, DDW
Dynamic View Alteration
Comments on Axiom Software DDW
Cross-Linkages with quite a few White Papers listed
Data Warehouse Configuration
Real Time Road Mapping
Percipio Tool for Dynamic Data Warehousing
ENTER GOOGLE SEARCH TERM: "Dynamic Data Warehouse"

Share: