|
« No one cares about your blog |
Main
| Providing ROI with Business Intelligence »
Tanjian Norman commented on a post from 2005 as follows:
"DW is dead? It seems that EAI/EII or more importantly service enablers like SOA are complementary to a DW environment. Please elaborate on why you think DW is positioned for replacement by EAI/EII technologies."
He was responding to my post in which I said, among other things: "DW will eventually go the way of EAI. The extra data store in the picture is redundant and the market eventually drives out inefficiencies."
I thought I would bring my response into a new post...
No, DW is not dead. It's alive and well and thriving. Almost every midsize and up company has at least a semblance of one. A robust data warehouse is highly desired and sought after everywhere. My point is about the future - when exactly I do not know. I agree that EII is complimentary-only to data warehousing today, but its merits should surely be considered.
Data warehousing is evolving. New data warehouses and data warehouse rearchitectures are well advised these days to consider not building a pure batch-loaded data warehouse where all analytic calculations, including master data and all reporting is done. There are several layers of calculations, functions and even data that are no longer necessarily part of a robust data warehouse reference architecture.
1. Master data calculations - Master data is not ideally calculated in a downstream data warehouse. It is needed in the operational environment as well as the data warehouse. As time goes on, the data warehouse will be a receiving system for the master data.
2. Operational business intelligence - I blogged about this here. There is certainly a lot of calculations that do not, or cannot, interact with the data warehouse in order to be effective. This can go well beyond basic operational reporting from a single system.
3. Yes, EII. EII is able to facilitate multi-system operational reporting and business intelligence. Some clients believe that several of their data sources do not need to be fed into the data warehouse if they can run EII queries and eliminate the redundancy of having the data in multiple places. EII can handle multiple databases in multiple formats, referential integrity, XML and basic transformations. EII still has a long way to go (query tuning, 2 phase commit, business metadata, memory constrained, etc.) and data warehouses are still absolutely vital, but it shows promise and is another factor chipping away at the data warehouse requirement.
4. Modern ERP - There was a time when ERP vendors debated that the data warehouse was not necessary. When this was proven untrue, they provided packaged data warehouses so at least they kept some of that business too. In the background, they've continued to add functionality to the base ERP to keep chipping away at the data warehouse requirements. Today, one of the striking things about ERP systems is they are keeping history data indefinitely. Having a historical data repository used to be one of the main reasons for building a data warehouse, but that is not always a data warehouse requirement any more.
And finally, a "real-time" data warehouse is evolving to look more and more like an ERP system itself, with real-time feeds of operational data, triggers and analytical applications. So, the definition of data warehousing may be changing to keep the term active, but the data warehouses themselves are evolving.
I hope this helps. Feedback welcome, which could be interesting...
|
Comments
Hi William
You make some interesting points in this post (and others within the blog). In particular I like the way that you are looking back at the evolution of data warehousing and projecting where it’s going, given business pressures, technology developments (including SOA and real time business intelligence technologies) and the changing expectations that people have.
This echoes may of the themes in my ebook “In Search of Insight” which can be downloaded free from http://www.seewhy.com.
This extends your discussion of both real time information and operational BI, both of which are significant trends for the industry.
In particular, we’re finding that these two are often connected. Why do you need real time information? Probably because there is a pressing operational problem that needs fixing. This of course leads directly to process, and an area that the DW struggles with. The data warehouse is a data centred store, not a process centred store. So wherever you need intra day information (typically problems or opportunities) presented in the context of a process state, the data warehouse is the wrong platform to achieve this.
The conclusion which quite a few people are coming to (analysts included) is not that the data warehouse is dead, merely that Business Intelligence will be done (and is already being done) much more broadly than solely centred around the data warehouse. Operational / process centric BI needs to have a very low latency store, and atomic level detailed data including process state. Latency is critical simply because you cannot automate a corrective action without real time data. Reporting that the process broke yesterday simply doesn’t cut it.
To address operational BI needs of this type, a different kind of architecture is required than a traditional data warehouse. As a result we see the emergence of a new category of BI technologies using Event Stream processing which are designed for this task.
These trends and technologies don’t mean that the data warehouse is dead, rather that its role is changing from ‘the place where you analyze data’ to ‘the system of record.’
Posted by: Charles Nicholls | September 8, 2006 3:57 AM
I want more details in data ware housing plz send more details regarding
Posted by: Ramesh.B | February 6, 2007 5:18 AM
I believe data warehousing is here to stay. It is the only place where you can keep track of your history and changes that has happened. Nothing in EAI/EII/ERP will ever replace that functionality.
I further believe that we are on the verge of building data warehouses that will outlast several ERP changes. Why? Well, for many it still comes as a nasty surprise that you will have to rebuild your warehouse every time you change one of the surrounding systems. Only in the last few years have we found methods to model our warehouses with the flexibility necessary to cope with changes like these.
If you are interested in the latest developments in data warehouse modeling, please look at our homepage: http://www.intellibis.se and read up on anchor modeling.
Posted by: Lars Rönnbäck | December 12, 2007 3:32 AM