I've posted and written many different things over the years about what technology (specifically BI tool vendors, and RDBMS vendors, and ELT / ETL vendors, EII, EAI vendors) need to have in the future. This is another look at an updated wish-list, along with market expectations and what I'm seeing as faults in the industry today. Don't get me wrong, the vendors (some) are scrambling to put new technology in place such as temperature based data, high speed interconnectivity, and massive parallelism to handle volumes - it's just they aren't quite there yet. So here's a look at what I'm hoping to see in 2007 and beyond.
Convergence, convergence, convergence - I've written about it, spoken about it, and conferred with colleagues about it. It's happening, like it or not. The once well-defined "niches" and edges that software and hardware technology vendors used to have are fading away. Customers want single consolidated instances of data, single points of management, and consistent (common) models, common architectures, common services, common metadata.... and they are (as always) attempting to reduce tool sets they use to move data around the organization.
So what does all this mean anyway? ETL/ ELT and RDBMS lines are blurring, EAI and EII, and Web Services lines are blurring, BI (reporting/analytic) and RDBMS lines are blurring, There's cross-over, cross out, buy-up, snatch up, use up cross company convergence happening. Look at HP - buys Knightsbridge, brings Tandem Non-Stop SQL back to life as MPP on HP SuperDome to compete with appliances. Appliances, coupling RDBMS, fast access, fast load paradigms, MPP and parallelism, and so on.
ELT and ETL are morphing into a balancing act between the RDBMS and "transform" in stream, but wait - there's more!! EAI has morphed (somewhat) into web-services, new vendors for web-services sprung up to handle metadata, and EII is playing in a middle-ware integration role. Each are vying for their space, but each are beginning to rely on the other for accessibility, transformation, on-demand information delivery, metadata management and so-on.
So what kinds of things should we expect to see in 2007?
I predict (which is hard to do) that ETL and ELT will become singular through tool utilization
* the last remaining independent tool vendors may be purchased by large hardware companies and the functionality morphed into RDBMS feature sets, and the rest of the functionality embedded in a web-services tool set built with SOA in mind.
* ETL and ELT will be around for a while as a legacy integration, but as people move more towards real-time or active, the need will dissipate somewhat - as systems finally consolidate, and OSS sets up single sources, common data models, and operational layers for SOA and the enterprise - legacy systems may finally sunset. This will trigger sunset on "moving large batches of history around".
* New systems that are written will be architected on a common data model that will act as BOTH an operational system, AND a data warehouse - these will no longer be separated.
* EAI will complete it's transition into the SOA space, and those vendors that don't - will eventually die out, as SaaS takes over applications, and web-services take over the integration components.
* MDaaS - metadata as a service - will begin to take shape. There will be new entries in the market place claiming common metadata for sale, linked to common data models - these models will shape the SOA market place and OSS / DSS systems of the future.
* EII will make it's mark in the metadata management world, and begin to truly allow business users point and click management of Ontologyâ€™s and Taxonomies of both DATA and METADATA, along with access paths, and security, and push-button web-services management.
* RDBMS vendors will push the barriers on volumes of data, adding compression of data sets, encryptions, searches of both encrypted and compressed data; new indexing mechanisms will arise as a result of DW2.0, volumes, and Real-Time or Active warehousing. RDBMS vendors will begin incorporating common data models as a part of their delivery to customers. RDBMS vendors will add in-memory aggregations, and temperature based data sets. RDBMS engines will perfect their self-tuning, and MPP operations, they will also perfect query re-writes, and provide dynamic aggregation capabilities for constantly accessed / grouped data.
* BI tool vendors will STOP having to deal with middle-server aggregations, and start having to address serious volume, serious performance. Those BI tools that don't run queries in parallel for the same report will have to re-write their core architecture to support "every query parallel - every time / all the time". BI Vendors will begin to push the envelope on Data Visualization, and exploration (walk-throughs of visual data). BI Vendors may even begin to experiment with visualizing data in new "models" that we haven't thought of, such as 3D (showing data as chemistry models for instance). BI Vendors will have to deal with scalability on single servers, they will have to make reports available via WEB-SERVICE REQUESTS, and at the selection of the web-service requestor will have to produce the report in any format requested.
* BI Vendors will finally have to deal with security at a cell level for display of sensitive data on the reports. Thresholds for individual fields will be set to hide data, and show data based on who, what and when.
There's more, if you're interested in hearing more of my thoughts, please don't hesitate to ask, however I'd like to hear from all of you - what is it you need most in 2007 from these vendors? What do you want them to produce for you? What isn't working for you right now?
Hope to hear from you,
CTO, Myers-Holum, Inc
Posted January 8, 2007 9:13 PM
Permalink | 2 Comments |