Blog: Dan E. Linstedt

« ETL & ELT - as they relate to VLDW | Main | Appliances for Business Intelligence and Data Warehousing »

Dynamic Nature of Systems

I've been out for a while; getting new companies started can be hard to do... But I think we're getting there. Anyhow, as everyone knows - I've got an interest in the real nature of things, how they work, how they change and adapt themselves, and how automated computing facilities can bring these systems closer and closer to the business users. Ultimately we'd like the business users to manage, and suggest change to the back-ends through uncomplicated interfaces. We have a lot of terms popping up on the internet (even just over the past two years) which I'd like to explore, and then we'll dive in to where I think the industry should be going. In particular, one term is driving me to sleepless nights: "Dynamic"

It seems to start and end with the term Dynamic. Throw the phrase on to the front of existing terms (like decision support, BI, DW, web services, etc...) and we have a whole new ball game. Just run a search in an engine like http://www.SurfWax.com - which uses a neural network to really find what you might be looking for... and see the hundreds of results that now pop up as a result.

I've been following some really interesting writing in ACM Communications journal, and IEEE on visualization, business tactics, and in their latest print volume, Dynamic Decision Support (caught my eye).

I'll try to summarize for you what they were getting at:
Dynamic Decision support (according to them) includes the notions of tying business process workflow to a visual design system on the front end, which allows the users full drag-and-drop access to their data, and to their metadata, and to the lineage - all connected within ontologies and sub-ontologies defined for each "workflow component." The business users would then design different workflows, and therefore re-organize the data sets underneath to produce their own reports, and their own "processes." In another article in this magazine, they continue on to discuss how the changes to these processes can be measured and managed in terms of impacts.

As you might imagine, there would be security around this entire process to keep them from blowing the larger business flows out of the water. However, they all live in this "world" and play with different levels of "process flow metadata", to help streamline and optimize the business itself. In other words, in their eyes, DSS is no longer just a reporting interface, nor is it actually developing dashboards, but an interactive process enriched with metadata, process analysis, and business workflows that manage dependencies, definitions, ontologies, and so on. The end result is getting the data out to a platform like "excel" or your favorite reporting tool - in an automated fashion.

Of course they also state that this direction is several years off, there's a lot of engineering to be done to ensure the quality, consistency, and usability of such interfaces. There is also a level of business education that must take place, along side of fundamental shifts in the way we do business, but another point they make is: that business must stay nimble, and in order to compete - businesses (even large ones) must be flexible and employees who do their daily jobs are the ones armed with the knowledge on how best to change the business for the future.

What does this have to do with DYNAMIC?
Everything. Too often this term is abused on the web - it's like the new buzz word used in front of all else. For instance: I went to lunch and ordered a dynamic pizza, or a dynamic menu that changes based on my dress style, or dynamic presentation, or dynamic speaker, or dynamic blog entry... You get the point, it's been tossed in front of Dynamic Data Warehouse, Dynamic Decision Support, Dynamic Reporting, Dynamic ScoreCards, Dynamic ETL, Dynamic Data Integration, Dynamic Web Services, Dynamic SOA.

But what does it _really_ mean?
In my humble opinion, the word "dynamic" should not be used lightly, yes there are standard definitions dating back to the Greeks, but I am of the opinion that when we use the word Dynamic in front of today's technologies we really are saying that structure (parts of metadata) change with the data and the processes. Well, at least that what I am suggesting. I contend that the nature of "dynamic systems" would expand to include "adaptive systems", to me: a dynamic system is one that enhances the user experience, and adapts to the user experience in a holistic fashion.

I think this is a paradigm shift, and there are different levels of getting to a Dynamic Data Warehouse, or a Dynamic Operational System, or even a Dynamic Enterprise Architecture. I think the levels are already built by standard ontological definitions of terms.

For instance: Dynamic Data Sets, Dynamic Information (a bit higher level), Dynamic Hardware, and so on. There is a difference in my mind between "active" and "dynamic." Again, I would take Dynamic Data Warehouse to encompass or include all the following components:
* Dynamic Data Sets
* Dynamic Hardware
* Dynamic Metadata
* Dynamic Structure
* Dynamic Technical Metadata
* Dynamic Business Metadata
* Dynamic Process Flows
* Dynamic Architecture
* Dynamic Reporting
* Dynamic Data Mining

These are the general components included at an architectural level in a non-dynamic or traditional Data Warehouse. They should (in my opinion) be defined as part of a Dynamic Data Warehouse. The same can be said for the term: "Real-Time" which I'd prefer to call "Right-Time", and the term "Active".

How do we define these components?
By defining each of the terms that are laid out with further definitional content I think we can walk further down the path. Dynamic to me, means capable of adapting to change in an automated fashion. Some of these terms are self-explanatory, but here are a few that I think have been missed by the industry:

Dynamic Data Warehouse - if we accept the fact that this term is a "complex system" which includes architecture, design, data, execution, software, and hardware, then we should also expect to see each of these components within the system to be "dynamic" and autonomous in their nature to change with the surrounding environment. A "Dynamic Data Warehouse" is _not_ (in my opinion) simply a Hardware Platform with a Database that accepts right-time data that might have a DSS thrown in.

There's a difference between providing Dynamic Access to right-time data, and Dynamic Data Warehousing in my opinion.

In my definition, the structure (metadata) must adapt, the business rules must be flexible and truly at the fingertips of the business users, those same rules must be part of the governance that helps the automated changes take place in the back-end, and so on. But if we look at a very simple example, let's see what a "Dynamic Data Model" might be:

Suppose I have an SOA in my organization, and at it's heart is a common data model, surrounded by a layer of common data services. These services "listen" to the outside world for transactions. When the transaction is ready, some of these services are smart enough to check the metadata (the version of the sending service, and the definition of the transaction itself). In doing so, the service discovers a new "XML" element; a new piece of information has been added. Now, in order to be a dynamic service, it must check the common model it has - and decide "what" to do with the new data, where does it put it? How does it adapt the model? Can it adapt the model? What’s the impact of the change if it were to make one? Does the service simply "stop" in mid-stream and fire an alert?

Well, the easiest answer is it takes the XML as a blob, and sticks it into an "XML aware" database object, theoretically it would send a workflow notification to the business users that this new element arrived. It would then be up to the business users to "adapt" or change their processes, and queries to make use of the element.

This is a very crude interpretation of a Dynamic Services component. What's missing is the automated adaptation of the element through MODEL DISCOVERY, META DATA MINING, and ARCHITECTURAL MINING with scoring, and "approximation" of context for where the new element fits, and where it impacts the process flows downstream. Not to mention the automated CHANGE to the process flows (when appropriate).

This begins to touch some of the components in a Dynamic System - but doesn't get to a Dynamic Warehouse. These are daunting tasks - and yes, we need to take baby steps along the way (and if someone can make a giant leap, all the better - if the market is ready). I would urge you to think of "Dynamic" in a new light, let's be more specific about WHAT it is we are creating that is dynamic rather than just throwing the label on an entire system.

I'm curious to hear your thoughts, as always - I like to take this as far beyond the horizon as possible. This is just one way to look at "Dynamic Systems" and "Dynamic Data Warehousing", if you have other ideas, please let me know.

Have a wonderful day,
Daniel Linstedt
http://www.COBICC.org (Colorado Business Intelligence Community Connection)

Share: