Tanjian Norman commented on a post from 2005 as follows:
"DW is dead? It seems that EAI/EII or more importantly service enablers like SOA are complementary to a DW environment. Please elaborate on why you think DW is positioned for replacement by EAI/EII technologies."
He was responding to my post in which I said, among other things: "DW will eventually go the way of EAI. The extra data store in the picture is redundant and the market eventually drives out inefficiencies."
I thought I would bring my response into a new post...
No, DW is not dead. It's alive and well and thriving. Almost every midsize and up company has at least a semblance of one. A robust data warehouse is highly desired and sought after everywhere. My point is about the future - when exactly I do not know. I agree that EII is complimentary-only to data warehousing today, but its merits should surely be considered.
Data warehousing is evolving. New data warehouses and data warehouse rearchitectures are well advised these days to consider not building a pure batch-loaded data warehouse where all analytic calculations, including master data and all reporting is done. There are several layers of calculations, functions and even data that are no longer necessarily part of a robust data warehouse reference architecture.
1. Master data calculations - Master data is not ideally calculated in a downstream data warehouse. It is needed in the operational environment as well as the data warehouse. As time goes on, the data warehouse will be a receiving system for the master data.
2. Operational business intelligence - I blogged about this here. There is certainly a lot of calculations that do not, or cannot, interact with the data warehouse in order to be effective. This can go well beyond basic operational reporting from a single system.
3. Yes, EII. EII is able to facilitate multi-system operational reporting and business intelligence. Some clients believe that several of their data sources do not need to be fed into the data warehouse if they can run EII queries and eliminate the redundancy of having the data in multiple places. EII can handle multiple databases in multiple formats, referential integrity, XML and basic transformations. EII still has a long way to go (query tuning, 2 phase commit, business metadata, memory constrained, etc.) and data warehouses are still absolutely vital, but it shows promise and is another factor chipping away at the data warehouse requirement.
4. Modern ERP - There was a time when ERP vendors debated that the data warehouse was not necessary. When this was proven untrue, they provided packaged data warehouses so at least they kept some of that business too. In the background, they've continued to add functionality to the base ERP to keep chipping away at the data warehouse requirements. Today, one of the striking things about ERP systems is they are keeping history data indefinitely. Having a historical data repository used to be one of the main reasons for building a data warehouse, but that is not always a data warehouse requirement any more.
And finally, a "real-time" data warehouse is evolving to look more and more like an ERP system itself, with real-time feeds of operational data, triggers and analytical applications. So, the definition of data warehousing may be changing to keep the term active, but the data warehouses themselves are evolving.
I hope this helps. Feedback welcome, which could be interesting...
Posted August 14, 2006 9:14 AM
Permalink | 2 Comments |