Operational BI (or Operational Reporting, the terms haven't really hardened yet) is a really important topic. The dialogue for the past decade and a half has been about data warehouses, loading data warehouses and using BI tools to report from data warehouses. It's become apparent that this is not a complete solution. Data warehouses typically (though not always) are designed to support after-the-fact reporting and analysis and the data models are oriented for that. Operational reporting is more immediate, more low level and more actionable.
There was an excellent webinar on BeyeNetwork today, sponsored by GoldenGate Software and delivered by Claudia Imhoff on this topic (an archived version should be available there by the time this blog is posted). Claudia explained the various options that exist (and the drawbacks of each), such as going directly against operational systems, using ODSs, re-architecting data warehouses to provide fresher data and more pliant data models and, presumably the most attractive solution, using various flavors and types of EII (Enterprise Information Integration) to dip into the data warehouse and the operational systems by creating some abstraction to allow data to appear as if were in a single location.
Why is Operational Reporting important? One reason is that the rise of business process modeling and execution systems will inevitably make business processes more dynamic and the data warehouse/BI structure will not be able to keep up. Pulling data directly from operational databases (or replicates or logs or even on-the-fly from message queues) avoids the steps of transformation and loading to a data warehouse, eliminating a lot of latency. In addition, data warehouse models are not operationally oriented, so its conceivable that in the transformation process, a lot of the semantics of the data at the operational level are lost. As a result, data created from Operational BI could not easily be written back to the operational systems.
Another reason is that BI has not been able to cross that "actionable" gap because its architecture is geared toward informing people, not operational systems. An Operational "reporting" system has to do more than just report, it has to react. That reaction can be driven by a human being (picking up the phone and sending the store manager to investigate a problem) or it can be driven by decision services as James Taylor and I described in Smart (Enough) Systems: How to Deliver Competitive Advantage by Automating Hidden Decisions.
Now that it is possible for business people to interactively modify running business processes with process automation tools, they will need help assessing what they've done. There won't be time to study the problem, devise some data models to support investigations and build data integration aplets. EII allows for a sort "canonical" map between the systems and analytical/reporting model so that these constant changes can be handled at a higher level of abstraction, almost instantaneously. By aligning the substantial resources of the BI industry to this effort, I'm pretty confident we'll make a lot progress quickly.
There was an excellent webinar on BeyeNetwork today, sponsored by GoldenGate Software and delivered by Claudia Imhoff on this topic (an archived version should be available there by the time this blog is posted). Claudia explained the various options that exist (and the drawbacks of each), such as going directly against operational systems, using ODSs, re-architecting data warehouses to provide fresher data and more pliant data models and, presumably the most attractive solution, using various flavors and types of EII (Enterprise Information Integration) to dip into the data warehouse and the operational systems by creating some abstraction to allow data to appear as if were in a single location.
Why is Operational Reporting important? One reason is that the rise of business process modeling and execution systems will inevitably make business processes more dynamic and the data warehouse/BI structure will not be able to keep up. Pulling data directly from operational databases (or replicates or logs or even on-the-fly from message queues) avoids the steps of transformation and loading to a data warehouse, eliminating a lot of latency. In addition, data warehouse models are not operationally oriented, so its conceivable that in the transformation process, a lot of the semantics of the data at the operational level are lost. As a result, data created from Operational BI could not easily be written back to the operational systems.
Another reason is that BI has not been able to cross that "actionable" gap because its architecture is geared toward informing people, not operational systems. An Operational "reporting" system has to do more than just report, it has to react. That reaction can be driven by a human being (picking up the phone and sending the store manager to investigate a problem) or it can be driven by decision services as James Taylor and I described in Smart (Enough) Systems: How to Deliver Competitive Advantage by Automating Hidden Decisions.
Now that it is possible for business people to interactively modify running business processes with process automation tools, they will need help assessing what they've done. There won't be time to study the problem, devise some data models to support investigations and build data integration aplets. EII allows for a sort "canonical" map between the systems and analytical/reporting model so that these constant changes can be handled at a higher level of abstraction, almost instantaneously. By aligning the substantial resources of the BI industry to this effort, I'm pretty confident we'll make a lot progress quickly.
Posted April 10, 2009 5:55 AM
Permalink | 3 Comments |
Neil -
Good stuff.
I'm concerned that Claudia was trying to get last-century architecture to provide for tomorrow's needs. It's like making my F350 Super Duty Dual-axle Turbo Diesel perform like a Prius.
But even in your discussion, you miss an obvious source of high-quality, real-time data: the Enterprise Service Bus. We should be subscribing to authoritative events, messages, state changes, and rules/policy violations directly off the message/service bus as they occur.
This eliminates (to the degree possible) the tight coupling, mass-data-replication, and brittleness that plagues existing BI architectures and gives us more agility thru autonomy.
Also, I believe operational reporting, while giving us more real-time reactions, should also enable real-time analytics that help us predict what may happen. Fraud analysis is an obvious use case. Tailored marketing is another.
Thanks for a great post!!!
Keep 'em coming!
Marty
Marty,
Great comments and I wish I could say that I thought about all of that but left it out for the sake of brevity, but that wouldn't be true.
I do disagree with your comment about Claudia (and though she has a Prius, she ain't no Prius). It's not my position to speculate on what a colleague is doing or why, but based on papers and presentations I've seen in the last few months, she is not relying on last-century architecture. For a long-time data warehouse stalwart to go public with the idea that there is life beyond the data warehouse is noteworthy.
I wasn't really precise about alternative ways to use operational data, but of course an ESB would be in the fight along with logs, queues, etc. Thanks for pointing it out.
I think I mentioned decisions. That would be impossible to do without analytics, which James and I described at length in the book.
-NR
Neil,
Nice blog,
Just a quick reaction. I disagree with the statement that BI has not been able to cross that actionable gap.
This is because we do not call it BI. We call it a Workflowsystem. Workflowsystems are build with automated or manual decision moments build into them. But also with realtime statistics about operational processes.
The question now is, is this BI? I think yes.
Can it be improved? Of cause!
Is it cost effective? Most of the time not.
Warning: Do not create realtime Operation BI just for BI sake. It is not worth the effort because it makes processes much more complex and thus much more expensive to maintain.
Guido Treur