Blog: Colin White

« Content Management Market Consolidates | Main | New CIO Spending Survey »

ODS, EDW, CDI, MDM: What's in a Name?

I read with interest Claudia Imhoff's recent blog about the ODS being alive and well.

http://www.b-eye-network.com/blogs/imhoff/archives/2006/08/the_ods_is_aliv.php

I agree! I have also heard statements about the ODS being dead and of course that's absurd. The problem is that people like to put labels on things and many of the data stores given the ODS label are work data sets or staging areas, and are not in fact ODSs.

In an ideal world we would have one data store that all applications use, and there would be no need for an ODS, or even data stores for data warehousing, CDI or MDM. Unfortunately, applications are developed at different times, have different data and data model requirements, performance needs, and so forth.

I remember when distributed database was the flavor of the day. Many of these efforts failed because of the heterogeneous data involved, and because of integrity and performance issues. Since then we have come up with a variety of data management and data integration options to try and overcome these issues. Like distributed database, these technologies work for certain types of application. What we have learnt from these efforts is that one size does not fill all when it comes to data management.

In general, there are five types of application processing in IT systems: business transaction, collaborative, business intelligence, planning, and master data processing. These applications may require current data or past data.

Most operational business transaction applications require current data. These applications and their associated data stores are dispersed throughout the enterprise. This dispersed data makes it difficult to gain a current view of enterprise-wide business operations. To solve this problem, required subsets of the operational data is integrated into an operational data store (ODS). The latency of the the data in the ODS will vary based on business needs. The ODS can be used for operational reporting, to feed downstream applications, and as a temporary bridge during legacy application migration.

Dispersed operational data also makes it difficult to gain a past view of enterprise-wide business operations. To solve this problem, the data warehouse concept was created. A data warehouse contains a historical subset of past data captured from the operational environment. The data sources for the data warehouse can be the operational applications themselves, or an ODS.

Collaborative applications are also dispersed throughout the enterprise. These applications require both current and past data. In this case, an enterprise view is obtained by capturing the dispersed data in a content management system (CMS). The data from the CMS data store can also act as a data source for a data warehouse. Planning data follows a similar pattern to collaborative data. The industry direction here is to consolidate planning data into a shared data store. Again this data store can be used as a data source for a data warehouse.

When data cannot be consolidated into a single data store for operational, BI or collaborative processing, then data federation approaches can help to create an integrated view of the dispersed data.

The last piece in the puzzle is master data processing. It is this type of processing that has sometimes led to the comment that the ODS is dead. Current master data is often intermingled with operational business transaction data. In these cases, master data finds its way in an ODS for operational processing, and into a data warehouse for BI and decision processing.

For ease of management, the direction of the industry is to separate master data from standard transaction data, and to manage both current and past master data in a separate integrated store. This approach removes current master data from business transaction systems and the ODS. It also removes past master data from the data warehouse. Standard transaction data, however, still flows into the ODS and the data warehouse.

It will take some time to get to full enterprise MDM. Meanwhile interim solutions like CDI hubs are being to used to propagate master data between systems. The key thing to note here is that these hubs only process master data. The remaining transaction data continues to flow into the ODS and data warehouse. If a CDI hub processes all the transaction data then it is performing the same identical function that an ODS does today, and there is no point in creating a new name for it.

Any comments?

Share: