We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

What Do SOA and ESB Mean in Business Intelligence?

Originally published June 19, 2006

There are several new and evolving enterprise integration and Web technologies that are likely to have a profound effect on the business intelligence (BI) marketplace. This month I want to discuss two of the more important ones: service-oriented architecture (SOA) and enterprise service bus (ESB).

Service-Oriented Architecture
A service-oriented architecture consists of a set of components that enable distributed resources (applications, database systems, etc.) to interact with each other in support of business processing. Organizations have been doing distributed business processing for many years, and an SOA is simply a way of formalizing this processing. In an SOA environment, resources are exposed as service providers. The interfaces to these providers are registered with a broker. Resources that want to consume services locate and interact with providers through the broker.

A key point to note is that an SOA can interconnect resources that operate at the user interaction, application and data levels of an IT system. In a data warehousing environment, for example, a data validation service could be defined as a service provider and called by data integration application. Other examples of services providers include user authentication, search, data transformation, BI analysis, data mining models, legacy applications and business transactions. To support an SOA environment, most business intelligence and data integration vendors are building service interfaces to their products.

The advantage of an SOA is that it enables common and shared interfaces to be defined and created for distributed resources. This eliminates the need to develop multiple proprietary point-to-point connections between resources, which reduces IT development and maintenance efforts and encourages service reuse.

A variety of different technologies can be used to build an SOA environment, define service interfaces and enable service brokers. One of the prevalent technologies is Web services, which has garnered industry acceptance because of the availability of open standards, vendor independence and product support. Web services are easier to implement than earlier SOA technologies, which are often proprietary and complex to use. Most of the SOA support coming from BI and data integration vendors is, in fact, based on Web services.

Enterprise Service Bus
Another important development in the SOA environment is an enterprise service bus (ESB). An ESB is a message-based capability that facilitates interaction between distributed resources. An ESB is not required for SOA, but it does increase the power and flexibility of SOA usage.

The objective of an ESB is to route messages between resources in a reliable manner; that is, it guarantees message delivery. Message routing may be done synchronously or asynchronously between source and target systems. Messages may be also be transformed from a source format into a target format as they pass through the bus. Other facilities offered by an ESB include load balancing and failover.

ESB middleware is available for both the Microsoft. NET and Java J2EE environments. This middleware may support a variety of SOA broker technologies such as Web services, Java JCA, Microsoft DCOM and CORBA. The advantages that an ESB brings to the SOA environment are security, reliability, scalability and the ability to interconnect older SOA broker technologies with Web services.

One of the potential users of an ESB is a data integration service. In fact, several vendors have modified their data integration and ETL (extract, transform and load) tools to be event-driven so that they can consume event messages from an ESB. These events can carry information about source data changes, which can be used by the data integration service to incrementally update an operational data store (ODS) or data warehouse. This approach is particularly useful for operational BI applications that require intra-day data.

When a data integration service is connected to an ESB, the question arises as to whether data transformation should be done in the ESB or in the data integration service. Some people even argue that an ESB service could be used to populate an ODS directly without the need for a data integration ETL tool. To answer this question, we need to examine how an ESB works and is used.

An ESB is used frequently to route transactions and data between operational applications, from a CRM application to an ERP application, or between B2B systems, for example. The data passing through the bus is usually reformatted to suit the needs of the target application. In some cases, the source data is transformed into a common XML format as it flows from the source system, through the ESB and to the target system. The advantage of a common format is that the same data can be consumed by several target applications and reformatted to suit the needs of each target. A message coming from a CRM system, for example, could be transformed into a common XML format, and then routed to both an ERP system and a data warehousing application. Common XML formats are also very useful when sending and receiving data from external systems.

Two things should be noted about potentially doing data transformation in an ESB. The first is that messages passing though the hub are usually routed in close to real time; and given the message volume involved, there is often not sufficient bandwidth to do significant data transformation and lookup. An ESB, therefore, is usually more suited to doing basic message reformatting than complex data transformation. The second point to note is that regardless of where data transformation is done, the business views, models and rules for doing that transformation should be common to all ESB and data integration applications.

It has always puzzled me why organizations focus on building shared business views and models of information in a data warehouse for use by BI tools, but don’t do the same thing for source data flowing into a data warehouse. A shared business view of source data (an invoice, for example) simplifies access to source data and can insulate data integration services from source data structure changes. A business view of the operational source data can be used in an ESB to map the source data into a common XML message, which can then be consumed by a data integration service. If the location or format of the source data is changed, then only the mapping in the ESB to the shared view needs to be updated – the data integration service is not affected.

Moving to an SOA Environment
I have just touched the surface of how a SOA and ESB can be used in data integration and business intelligence. Although SOA and ESB technologies are immature and still evolving, all BI users should have a plan to move to an SOA environment. This will be particularly important when building operational BI applications, which need to be tightly integrated with operational and collaborative systems. An SOA and ESB environment provides significant advantages for enabling this integration.

My next article will continue the discussion of the effect of new and evolving enterprise integration and Web technologies on business intelligence and data warehousing by looking at Web 2.0 and enterprise search.

  • Colin WhiteColin White

    Colin White is the founder of BI Research and president of DataBase Associates Inc. As an analyst, educator and writer, he is well known for his in-depth knowledge of data management, information integration, and business intelligence technologies and how they can be used for building the smart and agile business. With many years of IT experience, he has consulted for dozens of companies throughout the world and is a frequent speaker at leading IT events. Colin has written numerous articles and papers on deploying new and evolving information technologies for business benefit and is a regular contributor to several leading print- and web-based industry journals. For ten years he was the conference chair of the Shared Insights Portals, Content Management, and Collaboration conference. He was also the conference director of the DB/EXPO trade show and conference.

    Editor's Note: More articles and resources are available in Colin's BeyeNETWORK Expert Channel. Be sure to visit today!

Recent articles by Colin White



Want to post a comment? Login or become a member today!

Be the first to comment!