We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Business Intelligence and Data Delivery: Converging with SOA

Originally published November 13, 2007

There are two main technical trends I’m seeing in my clients today. The first is their drive for information that has integrity, consistency, and usability to satisfy a wide range of uses, including enterprise-wide reporting, executive dashboards, operational analytics, data mining, and ad hoc reporting and queries. The second trend is toward using service-oriented architecture (SOA) to create a more responsive and agile application environment to increase reusability and reduce duplication of software functions, such as capturing customer information and their associated data. Both trends intend to help businesspeople and applications adapt to new business needs more easily.

Because data and processing cannot be truly separated, these two trends are converging. Also, as business intelligence (BI) becomes near real-time, it is becoming more integrated into the application environment. This integration can be accomplished using an SOA infrastructure for messaging and/or data transport. As SOA becomes a foundation for application systems, the demands on database design need to be thought through because it differs from those of traditional stand-alone applications.

There is, as there has always been, a tension between the two: data and its processing. Both disciplines are essential in order to establish an architectural framework that will meet the application function and data delivery needs of the organization. This tension is not naturally reconcilable. How the BI/SOA convergence is visualized will set the direction for applications and data delivery in an organization, for better or for worse.

As an example of this challenge, ask an SOA architect to draw a conceptual diagram for SOA. Almost universally, data will be drawn at or near the bottom and data issues are rarely discussed; all the interest is in the SOA components. When I ask a BI and data delivery architect to do the same, it is applications that are drawn at or near the bottom and data issues become preeminent while application issues are rarely discussed.

The Discipline of SOA

SOA seeks to take application functions and decompose them to their smallest units of work, the unit that cannot be usefully decomposed further. At this level, data is handled in discrete pieces, often at a data element level. At its core, SOA is disintegrative.

SOA will be best enabled through a single, highly normalized, enterprise transaction data store. The standard SOA vision, simplifying access to disparate existing application data stores, is problematic. It fails to address the traditional challenges of keeping data that is duplicated and maintained in several places synchronized and consistent with other related data in all the places the data resides. Building services to do this will be quite complex, considering the number of directions that data changes will likely come from. Of these two approaches, the single data store is sounder. I will come back to this later.

The Discipline of BI and Data Delivery

From the outside, business intelligence may appear to be simply a duplicate data store used to deliver reports and queries. As data professionals know, data elements by themselves have limited value. The work here is to bring related data elements together, integrate them, and organize them to be easy to use by businesspeople. At its core, BI and data delivery is integrative.

The challenge of data is that it needs to be dynamic and free-flowing. Business intelligence and data delivery is not simply about creating reports and queries. It is reconfiguring, recombining, and restructuring data as well. It requires delivering data in new ways, new forms, and always to the specific needs of users. Where applications and services strive for stability, BI and data delivery deals with fluidity and reconfiguration. This is why BI and reporting platforms are effective vehicles: they provide platforms to manage data, reports, queries, users, user rights and security, and access controls. This technology complements SOA and should not be viewed as replaceable by it.

The Nexus of BI and Data Delivery and SOA

The single, highly normalized, enterprise transaction data store is where BI and data delivery and SOA converge. With this data store in place, an organization and its application functionality would have no redundant, inconsistent, or non-integrated data. The enterprise data model, long desired but never before implemented, would be a physical reality embodied by this data store.

Setting aside for the moment the assumption that an organization can achieve this single, highly normalized, enterprise transaction data store, this data store on its own would not be suitable for business intelligence and data delivery. Yes, it would contain all the data and data rules used by the business, but it would not be easy for business users to use. Further, the data store would not likely contain the metrics and supporting drill-down, roll-up, and other analytic features.

While this data store, in the minds of SOA architects, would serve all application processing and reporting needs and eliminate the need for data warehouses, data marts, and BI, in reality it is deficient in analytic support but would replace the need for data integration and transformation.

The Framework for a Successful BI/SOA Convergence

How then should the BI/SOA convergence be visualized? Every organization should diagram a reference architecture that shows the critical objects, relationships, and associated technologies within the IT environment, including legacy and emerging technologies. Looking solely at SOA and BI and data delivery, this reference architecture should include the framework for:

  • From an SOA perspective, the services, services management, and services integration broker components should be included;

  • From a BI and data delivery perspective, the enterprise reporting, dashboard and metrics, data quality management, data integration, master data, and enterprise data warehouse components should be included;

  • From a convergence perspective, there are critical areas where SOA and BI and data delivery come together: (1) the meta-libraries for metadata and meta-processes; (2) the business rules engine repository of data and application rules, and (3) the enterprise data structures of the single, highly normalized, enterprise transaction data store and, where this is not the only application data store, the organization’s master data.

There has always been a tension between process and data, and SOA and BI are no different. To be successful, the proper vision of how SOA and BI complement each other and how they can be managed successfully must be developed and a reference architecture produced. Without one, one or both of these technologies can be misused and impair the IT organization and its support of the business.

Let me know what you’re thinking – e-mail at strategicthinking@rcgit.com.

  • Richard SkriletzRichard Skriletz

    Richard is a manager and management consultant with more than 35 years experience working in large corporate and start-up environments. His professional focus is on the strategic application of information technology to improving operational performance, managing organizational and technical change, and optimizing business effectiveness. Richard is a Global Managing Principal with RCG Global Services and CEO of InfoNovus Technologies. He can be reached via email at Richard.Skriletz@rcggs.com.

    Editor's Note: You will find more articles and resources, and Richard's blog in his BeyeNETWORK Expert Channel. Be sure to visit today!

Recent articles by Richard Skriletz

 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!