Despite the issue of lack of funding, 36 percent of organizations said they devote a large amount of staffing and budget to data integration. This will increase to 55 percent within 18 months. As a matter of interest, IDC estimates that data integration spending worldwide will increase from $9.3 billion in 2003 to $13.6 billion in 2008. IDC also says that the average enterprise spends 40% of its IT budget on data integration. This latter figure is similar to the TDWI result. The TDWI data integration report and Webcast will be presented in October

It was quite clear data integration is a hot topic and this theme dominated the Northwest BI Summit. The field is awash with acronyms, however. Not only do we have EAI, ETL, and EII, but also EDR, CDC, ECM, MDM and CDI.

On the second day of the summit, William McKnight led a discussion on Master Data Management (MDM) and Jill Dyche on Customer Data Integration (CDI). Both the TDWI study and the summit showed we still don’t have nice crisp definitions for these latter two areas. To generate some discussion, I wrote a piece in this month’s business integration newsletter on MDM and CDI. I am interested to hear your comments on MDM and CDI, and data integration in general.

Organizations having been integrating their customer data for some time and there are a variety of different ways of doing this. In the business intelligence and data warehousing space this is often done by using data integration software (e.g., ETL tools) to capture customer data from operational source systems and consolidate it in an operational data store (ODS).

In the operational business transaction area, companies like SAP have been using master data management (MDM) techniques to manage customer data. MDM solutions usually offer the ability to asynchronously consolidate data in a single customer data store (as in an ODS), or to synchronously propagate data changes between business transaction applications.

It is important in MDM and CDI solutions to separate the business application from the underlying technology. MDM and CDI applications are focused on providing a single view of master data (customers, products, etc) to business users. These applications document the business semantics of the data, provide data quality and data analysis facilities, and so forth. Underneath these these applications are various data integration technologies that can propagate, federate, or consolidate master data based on business need. Often the data stores created and managed by these business applications and technologies act as a data source for data warehouses.

MDM and CDI applications may provide analytical capabilities, but in essence they are primarily operational, rather than business intelligence applications. However, they do make use of data warehouse integration technology. This shows the industry direction toward data integration being considered at an enterprise level, rather than just at a data warehousing level. This of course has implications for IT organizational structures.

The direction of the industry then is toward MDM operational applications that manage, report on and analyze master data. The primary focus of these applications at present is on customer data, hence the CDI buzzword. These applications provide data quality and metadata capabilities that are focused toward the business area being addressed. They use a common set of services to manage and integrate data. These services may use a variety of data integration techniques and technologies to support both virtual and physical data stores. These data stores in turn can act as an integrated data source for data warehousing.

The acquisition of DWI by IBM is consistent with its thrust toward supporting enterprise-wide data integration and master data management technologies and applications. This will put pressure on the applications vendors to compete at both an application and technology level. This pressure will be particularly intense for Siebel who must claim ownership of the CDI space if it is to survive and grow.