Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

I've posted and written many different things over the years about what technology (specifically BI tool vendors, and RDBMS vendors, and ELT / ETL vendors, EII, EAI vendors) need to have in the future. This is another look at an updated wish-list, along with market expectations and what I'm seeing as faults in the industry today. Don't get me wrong, the vendors (some) are scrambling to put new technology in place such as temperature based data, high speed interconnectivity, and massive parallelism to handle volumes - it's just they aren't quite there yet. So here's a look at what I'm hoping to see in 2007 and beyond.

Convergence, convergence, convergence - I've written about it, spoken about it, and conferred with colleagues about it. It's happening, like it or not. The once well-defined "niches" and edges that software and hardware technology vendors used to have are fading away. Customers want single consolidated instances of data, single points of management, and consistent (common) models, common architectures, common services, common metadata.... and they are (as always) attempting to reduce tool sets they use to move data around the organization.

So what does all this mean anyway? ETL/ ELT and RDBMS lines are blurring, EAI and EII, and Web Services lines are blurring, BI (reporting/analytic) and RDBMS lines are blurring, There's cross-over, cross out, buy-up, snatch up, use up cross company convergence happening. Look at HP - buys Knightsbridge, brings Tandem Non-Stop SQL back to life as MPP on HP SuperDome to compete with appliances. Appliances, coupling RDBMS, fast access, fast load paradigms, MPP and parallelism, and so on.

ELT and ETL are morphing into a balancing act between the RDBMS and "transform" in stream, but wait - there's more!! EAI has morphed (somewhat) into web-services, new vendors for web-services sprung up to handle metadata, and EII is playing in a middle-ware integration role. Each are vying for their space, but each are beginning to rely on the other for accessibility, transformation, on-demand information delivery, metadata management and so-on.

So what kinds of things should we expect to see in 2007?
I predict (which is hard to do) that ETL and ELT will become singular through tool utilization
* the last remaining independent tool vendors may be purchased by large hardware companies and the functionality morphed into RDBMS feature sets, and the rest of the functionality embedded in a web-services tool set built with SOA in mind.
* ETL and ELT will be around for a while as a legacy integration, but as people move more towards real-time or active, the need will dissipate somewhat - as systems finally consolidate, and OSS sets up single sources, common data models, and operational layers for SOA and the enterprise - legacy systems may finally sunset. This will trigger sunset on "moving large batches of history around".
* New systems that are written will be architected on a common data model that will act as BOTH an operational system, AND a data warehouse - these will no longer be separated.
* EAI will complete it's transition into the SOA space, and those vendors that don't - will eventually die out, as SaaS takes over applications, and web-services take over the integration components.
* MDaaS - metadata as a service - will begin to take shape. There will be new entries in the market place claiming common metadata for sale, linked to common data models - these models will shape the SOA market place and OSS / DSS systems of the future.
* EII will make it's mark in the metadata management world, and begin to truly allow business users point and click management of Ontology’s and Taxonomies of both DATA and METADATA, along with access paths, and security, and push-button web-services management.
* RDBMS vendors will push the barriers on volumes of data, adding compression of data sets, encryptions, searches of both encrypted and compressed data; new indexing mechanisms will arise as a result of DW2.0, volumes, and Real-Time or Active warehousing. RDBMS vendors will begin incorporating common data models as a part of their delivery to customers. RDBMS vendors will add in-memory aggregations, and temperature based data sets. RDBMS engines will perfect their self-tuning, and MPP operations, they will also perfect query re-writes, and provide dynamic aggregation capabilities for constantly accessed / grouped data.
* BI tool vendors will STOP having to deal with middle-server aggregations, and start having to address serious volume, serious performance. Those BI tools that don't run queries in parallel for the same report will have to re-write their core architecture to support "every query parallel - every time / all the time". BI Vendors will begin to push the envelope on Data Visualization, and exploration (walk-throughs of visual data). BI Vendors may even begin to experiment with visualizing data in new "models" that we haven't thought of, such as 3D (showing data as chemistry models for instance). BI Vendors will have to deal with scalability on single servers, they will have to make reports available via WEB-SERVICE REQUESTS, and at the selection of the web-service requestor will have to produce the report in any format requested.
* BI Vendors will finally have to deal with security at a cell level for display of sensitive data on the reports. Thresholds for individual fields will be set to hide data, and show data based on who, what and when.

There's more, if you're interested in hearing more of my thoughts, please don't hesitate to ask, however I'd like to hear from all of you - what is it you need most in 2007 from these vendors? What do you want them to produce for you? What isn't working for you right now?

Hope to hear from you,
Dan Linstedt
CTO, Myers-Holum, Inc
http://www.MyersHolum.com


Posted January 8, 2007 9:13 PM
Permalink | 2 Comments |

2 Comments

Hi Dan,

I live in Brazil. Nevertheless I attended to your very interesting speech at DW 2.0 Conference, Anaheim,CA in November, 2006.

Congratulations for Data Vault Model and the approach you took on this wish list for BI of tomorrow!!!

Hi Dan,
I too share that same dream, one that would make the BI process more portable, pluggable and readily exportable into any format.
Great Wish List

Leave a comment

    
Search this blog
Categories ›
Archives ›
Recent Entries ›