Business Intelligence Network - The Vision for BI and Beyond

In each episode of "This Old House,” the homeowner, contractor and camera crew puzzle over how they are going to solve a difficult construction problem. Invariably, ever-at-the-ready Norm pulls out the right tool, and like magic, the problem is solved.

When it comes to delivering operational business intelligence (BI) to business users, the right tool that “Norm” might recommend is data virtualization. Here’s why, and how it works. 

Operational Business Intelligence – Valued, but Adoption is Slow

The value of operational business intelligence (BI) as a tool to help manage and optimize operational or time-sensitive business processes is well recognized. According to a recent survey by Ventana Research¹, 70 percent of executive management responders view operational BI as very important, citing greater efficiency, improved customer satisfaction and better access to information as top benefits. However, according to a recent survey by The Data Warehousing Institute (TDWI)², only 53 percent of respondents claim support for operational BI, and among those, only 16 percent claim to have a fully functioning, operational BI environment.

It appears most firms are stuck in early-stage adoption, in which they use daily operational reports to analyze various sales, finance, marketing, service and other business processes. Few have advanced to higher value operational activities such as process monitoring and execution.

What inhibits this advancement? Data.

Data Integration Stalls Operational Business Intelligence

Operational BI requires up-to-date data from multiple sources – data that can be accessed, combined and delivered rapidly as end-user reports and dashboards, or as alerts and triggers within the automated business processes themselves. Challenges to achieving this include:

• Multiple Silos, Disparate Data: Packaged ERP, SCM, HRMS and CRM applications combine with a variety of legacy and custom applications forming multiple silos of transactions and master data. Additional replicated sources such as operational data stores (ODSs), data marts and data warehouses add to the complexity. Each source has its own access mechanisms, syntax, security and more. Few are structured properly for reuse. Integrating data across these silos is beyond most standard packaged application reporting tools.
  
  
• Data Latency: Data latency must be reduced to move from reactive, operational analysis to proactive, operational optimization and execution. This is a delicate balancing act. Real-time data access must not come at the expense of online transaction processing (OLTP) performance.
  
  
• Data Replication: In prior technology generations, when processor and network performance was lower, approaches developed during this time typically relied on batch ETL integration technology to replicate transaction data into ODSs, marts and warehouses. While simplifying downstream reporting, this replication comes at a cost, particularly in terms of the additional long-term support and controls required for all the duplicated data.
  
  
• Data Agility: Business processes and operational BI needs are not static. Competitive pressures, new product introductions, continuous improvement initiatives and more force continuous change to information needs.

Why not Stick with ETL and Operational Data Stores?

Traditionally, enterprises have used ETL data integration technology with replicated data stores to address their data challenges because these tools are proficient in integrating across data silos. With innovations such as changed data capture and trickle feeds, data latency can often be successfully addressed, but typically at a higher cost for such items as development, due to the extra time required to develop ETL scripts and harden the physical ODS. There are also ongoing IT operational costs required to store and manage replicated data. Opportunity costs may be even higher and are especially critical in today’s high-change operations. Longer development means fewer projects are completed, delaying the realization of operational BI business benefits.

Data Virtualization Provides an Alternative

Data virtualization has proven successful in overcoming these challenges, while accelerating operational BI initiatives and lowering overall costs. Enabled by a middleware known variously as distributed query, virtual data federation or enterprise information integration (EII), today’s solutions provide three key capabilities:

• Data virtualization serves up data as if it is available from one virtual operational data store, regardless of how it is physically distributed across data silos. Query optimization and caching enable the high performance required to meet latency objectives without replication.
  
  
• Data abstraction simplifies complex data by transforming it from its native structure and syntax into reusable views and web services that are easy for operational BI solutions developers to understand and operational BI solutions to consume. Common higher level abstractions might include customers, invoices, shipments, payments and more.
  
  
• Data federation securely accesses diverse operational and historical data, combining it into more complete and meaningful information for a range of operational BI uses.

At build time, data virtualization provides an easy-to-use data modeler and code generator that leverage metadata to create abstracted relational views or web data services from source data.

At run time, the operational BI consumer views data as part of a virtual, operational data store. When the operational BI report or process needs data, the data virtualization middleware executes high-performance queries that securely access, federate, transform and deliver this data in real time.

By avoiding physical data replication in an ODS, development time and ongoing costs are reduced. Further, as requirements change or expand, modifying the models and regenerating the data services can be completed in minutes, without requiring IT resources to physically rebuild the ODS. Both the business and IT share in these benefits.

Data Virtualization in Action

Data virtualization enables a range of operational BI functions from simple operational reporting to complex process execution. One financial industry firm recently used data virtualization to cover both extremes for its mortgage loan business process.

Jumpstarting Stalled Operational BI

While operational BI projects are arguably more challenging than average home repairs, having the right tool for the job is equally critical in both cases. Surveys show that operational BI adoption is stalled, preventing the realization of its anticipated benefits. At the core of the problem is data, with its multiple silos, latency, redundancy and the need for agility. ETL and ODS have helped, but at higher associated costs than many enterprises can afford. Data virtualization helps to overcome data integration challenges by jumpstarting stalled operational BI initiatives and accelerating returns. With the right tool, problem solved.

References:

1. Ventana Spotlight: Operational BI Trends, Copyright 2007, Ventana Research.
2. Best Practices in Operational BI – Converging Analytical and Operational Processes, TDWI Best Practices Report, Wayne W. Eckerson, Third Quarter 2007, Copyright 2007, 1105 Media Inc.