It is interesting that whenever we talk about business intelligence we immediately think of it in the context of data warehousing. The two always go together, right? Wrong! I think we have been indoctrinated into thinking this way. We have lost sight of the fact that data warehousing only came about because we couldnâ€™t design our operational systems right in the first place.
This point was brought home to me while teaching a class on operational BI with Claudia Imhoff at the recent DAMA conference. It became obvious that some people came into the seminar thinking of operational BI in terms of data warehousing. When I started talking about the concept that operational BI should be process driven and tightly integrated with (and possibly embedded in) operational processes, it came as somewhat of a surprise to some attendees that data warehousing didnâ€™t appear in the picture.
I also discussed master data management (MDM) on the seminar. It would be reasonable to ask what MDM has got to do with operational BI. Well the problem is that some so-called operational BI applications are not really BI applications, they are MDM applications. An example would be creating a single view of the customer. This has nothing to do with BI. Itâ€™s an operational issue, not a decision support one. This doesnâ€™t mean to say that integrated customer master data cannot be used in business intelligence processing.
Hub products that frantically move data into an operational data store or data warehouse to create a single view of something add latency to the data, and are simply papering over the cracks of master data problems, rather than trying to solve the issue at the source, i.e., in the operational systems.
We should be aiming to put less data into the data warehousing environment, not more. Our long term objective should be to eliminate the data warehouse. The first step in this process is to remove master data from both operational transaction systems and from the data warehouse. This data should be stored in a separate master data store (MDS) that is maintained by separate master data applications. We can start to do this as we redesign our operational systems and move to a services-oriented architecture. The MDM system simply becomes a service. In this scheme, the MDS contains both current and historical master data. Both operational and BI applications can access the master data by calling the MDM system as a service. Initially, for performance reasons, business view subsets of the master data may be replicated into a data warehouse to act as dimension tables.
Removing master data from the data warehouse has the advantage of also removing much of the complexity and many of the data quality problems from data warehouse design. A separate MDM environment also simplifies the operational data store (ODS). The ODS now only needs to contain integrated business transaction data. The ODS effectively becomes an operational transaction data store, or OTDS.
The concept of splitting an ODS into an MDS and an OTDS was well accepted by many of the seminar attendees. I have seen several companies in both the US and Europe do this. In some cases the decision support environment consists of an OTDS, MDS and data marts, i.e., there is no enterprise data warehouse. Heresy, I hear people say. My answer is that we have to start thinking outside of the box. For example, there are some very viable search technologies appearing that also enable organizations to build BI applications without the need for an data warehouse.
Some of my comments above are a little tongue in cheek. The objective is to get people to accept that a data warehouse is not always required for business intelligence, and as I said earlier, the long-term objective should be to eliminate the data warehouse. Comments?
Posted March 22, 2007 1:34 AM
Permalink | 8 Comments |