We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

In recent posts I have begun to discuss a notion, or concept regarding something I call Dynamic Data Warehousing. It's real name should be Dynamic Structural Change and Adaptation Data Warehousing - but who would buy that? Not very marketing like if I say so myself.

I recently blogged on the war of the appliance vendors, and have written articles in the past on Convergence, and the wave of integration and partnerships sweeping the industry. This is just one of the futuristic items that I believe is completely possible to build with today's technology.

Today it may be expensive, but it can be done. In the future as the market makes way for more consolidations and integrations or partnerships between hardware and software vendors we will see additional efforts headed toward automatic structural manipulation.

I also added an entry on 3-D modeling capacities, and if the DDW device ever is produced, attaching a 3-D modeling landscape would increase the value 50 fold or more. So what are the basics needed for DDW "device" or appliance?

Here's a partial list of architectural components:
1. High speed hardware with embedded RDBMS software (embedded into the firmware), such that model changes and data movement is quick and painless, indexing is not needed.
2. Back-plane with several card-slots.
3. Each of the slots should be taken by one of the following cards: Data Mining/Structure Mining, Security and Access Card, Data Access - web-browser card, SOA and web-services card, Real-Time and Batch data integration card.
4. If I had my choice, one more card: chemical modeling software retrofitted to represent data, and data clusters - so we can visualize the information in 3-D format.

Each particular card has a job to do, but they all talk to each other through the backplane - high speed bus transfer. Data never leaves disk, or is buffered in high speed RAM.

The dynamic section of this device is to use Neural Nets, and data mining capacities across the structural components - to explore and dynamically adapt to newly arriving business elements. What we want from an IT perspective is the ability to plug and play a system, as we feed it data - it "discovers" the inherant structure, we teach it to model, and give it rules for performance and interaction with the existing storage device, CPU's and RAM that the card is plugged into. We fine tune the neural net over time, and it becomes a highly responsive processing machine that knows the "what" portion of our business.

From the business side, we make the structural inferences with confidence ratings, put the models back in the hands of business through BI and 3-D modeling efforts, bring the storage of the data, the model, and the presentation layers up to meet the business processes - closing the gap between IT and business. Train the business users (a few who wish to perform the tasks), and give them full reign over exploring the data sets through a 3-D landscape.

From a structure perspective, we adapt, change, and alter the structure through the structural neural net - tweaking for the balance between performance and business representation.

And finally from an information quality perspective, the appliance helps with both the structural improvements, as well as the data improvements through a second neural net - that imputes values, standardizes, cleanses, and reports metadata (the results).

If packaged properly, this device can rapidly become "smart" - no, not think for itself, but it will begin to lower costs, lower overhead, allow "what-if" games to be played with the architecture and judge the impact (visually).

Dynamic Data Warehousing is just beginning, I hope to hear back from you on your thoughts.

Dan L

Posted July 15, 2005 11:29 PM
Permalink | No Comments |

Leave a comment

Search this blog
Categories ›
Archives ›
Recent Entries ›