Common Metadata The Foundation Stone for Intelligent Business

Originally published July 18, 2006

Editor’s note: This article was originally published at http://www.businessintelligence.com/ex/asp/code.46/xe/article.htm.

What is Intelligent Business?
Intelligent business is a fundamental shift in thinking for the world of data warehousing and business intelligence(BI). For years, we have spoken about operational systems over here and analytical ones over there. There has been a “great divide” between the two camps with IT people who work on either side of the divide not even communicating with each other, let alone knowing what the other is working on or communicating with business. Intelligent business is about is taking business intelligence and putting it at the very heart of the enterprise. This idea here is that so called “traditional” data warehousing and business intelligence continues as normal (e.g., business analysts and managers reporting and analysing data using business intelligence tools and analytic applications); but, in addition, operational applications (executing as part of a business process) and portals can request trusted business intelligence on demand. It is like saying that business intelligence guides operations or that operations encapsulates business intelligence so that we have “intelligence inside” every operational activity that needs it.

Intelligent business is about integrating on-demand business intelligence into operational business processes whereby the business intelligence system responds to requests for just-in-time intelligence. On-demand business intelligence may or may not need to be integrated with other operational data on the fly before delivering this data to applications. Intelligent business also includes on-demand requests for recommendations that rely on the use of predictive analysis, for example. The point here is that business intelligence is not targeted at people such as business analysts, but is instead targeted at applications within a business process. This concept challenges the traditional approach to business intelligence because it removes the “great divide” between operational and analytical systems so that these systems communicate with each other constantly. The objective is clear, ubiquitous business intelligence in every activity in every business process across the enterprise so as to guide business operations toward achieving strategic business objectives. Hence, intelligent business uses business intelligence as an on-demand resource to guide core business operations as the company operates. In most cases, the operational business user doesn’t know they are using business intelligence. In other words, business intelligence is just there, transparent but integrated into normal operational applications. Behind the scenes, business intelligence Web services make it possible to dynamically integrate with operational systems. In this case, intelligence is used to guide people and systems in the context of the business process activity being performed such that operations are steered toward meeting strategic objectives and goals. In that sense, intelligent business requires that a business supports:

  • On-demand requests for specific intelligence (e.g., about a specific customer).
  • On-demand requests for automatic analysis (done on behalf of users) of data, rule-driven automatic alerts and automatic recommendations.
  • Automatic capturing of events in business operations that trigger the integration of other data on-demand, to be automatically analysed and manual or automatic actions taken. This is known as business activity monitoring (BAM).

All this requires that business intelligence is used as an intelligent resource for on-demand processing in the context of an operational business task being performed in a business process. Therefore, business process activities have to be “attached” to business objectives and goals declared in corporate performance management (CPM) software to help a company align business processes with its business objectives. Once this is done, we can then associate or target business intelligence at specific business process activities so that the business intelligence is used in an operational context to achieve a strategic business objective. At the moment, we are finding it difficult just to integrate corporate performance management and business intelligence, let alone integrate corporate performance management and business process management software or target business intelligence at a business process activity. Nevertheless, these requirements are critical to intelligent business.

Given this description, it is clear that an intelligent business strategy must be based on common trusted intelligence. Also, this vision says that the number of requests for business intelligence from people (using business intelligence tools and analytic applications for example) is likely to be dwarfed by the number of requests coming from operational applications being used in front-line operations by employees, partners, suppliers or customers. The vast majority of people in these roles don’t have time to use business intelligence tools because their job function is real-time. People in these operational roles spend the vast majority of their days in transaction processing systems or other manual activities. These people could be customer-facing staff, including customer service representatives in call centres, banking staff in branches at counters, mobile workers such as sales and field service, people in distribution, people in procurement and even finance people. Figure 1 shows the concept of intelligent business.


Figure 1

Notice in Figure 1 that operational applications surround rule-driven business intelligence services that access consistent integrated data in data warehouses and data marts. Also, the corporate performance management (CPM) software is not only integrated with business intelligence but also integrated with business processes so that we know what process activities are associated with what strategic objectives. Through the enterprise portal, people see their alerts, their recommendations, their actions – the on-demand business intelligence that is relevant to their role in the context of any business process activity they are performing at that time. They also see the collaboration tools that they need to do their jobs and collaborate with others.

The Need for Common Metadata
Having painted the intelligent business picture that many companies seem to want, there is one foundation stone that must be laid first if this vision is to ever become a reality. That foundation stone has to be common business metadata. By that, I mean common data names, common data definitions and common data integrity rules such that all data items in dimensions and metrics used in a data warehousing system are absolutely consistent. If there is one prerequisite to integrating business intelligence into operational business processes, it has to be that the business intelligence is trustworthy and has common understanding – trusted metrics, trusted meaning, trusted unambiguous data names and an audit trail of how data got to be in the data warehouse and/or the data marts that store it. Without this, integrating business intelligence into operations could cause untold damage to business operations, especially if automated analysis and actions cause business changes to be made.

What is common metadata? This is the very basis of data sharing. It is common definitions of data items, common data names, common integrity rules, common mappings and common transformations for:

  • all master data (customer, employee, product, location, etc.) items – whether used in operational systems or in business intelligence systems as dimension data,
  • all business transaction data, and
  • all business intelligence system metrics.

These “official” and common data items (sometimes called a shared business vocabulary – SBV) can be used not only in business intelligence systems, but also in operational systems. The point here is that for business intelligence to be at the centre of enterprise operations, with automated alerts, recommendations and action messages being fired on detection of exceptions, then that business intelligence had better be correct and commonly understood. So the prerequisite for intelligent business from a business intelligence perspective is to resolve all inconsistencies in data naming, data definitions and data integrity rules across all business intelligence systems by making these systems adhere to common data definitions in a shared business vocabulary.

People must resolve these discrepancies in logical data models, physical data models, business intelligence tool business views (e.g., Business Objects universes, SAS Information Maps, SQL Server report models), etc., if there is ever to be success here. To quote Michael Brackett, “building a common data architecture may seem expensive, but not building one is even more expensive” (Data Sharing – Using A Common Data Architecture, ISBN 0-471-30993-1).

Thus, a shared business vocabulary is needed. The benefits of a shared business vocabulary include:

  • A better understanding of data
  • Increased awareness and reduction of uncertainty about data
  • Formally named, comprehensively defined, properly structured and high quality data
  • The delivery of incremental short-term benefits while moving toward a long-term goal
  • Reduction of volume of data via removal of redundant data
  • Reduction of need for transformations if data is common
  • Improvement in productivity as people spend less time struggling to understand data and more time using it to do their jobs

Therefore, the very first step involved in intelligent business is to put a shared business vocabulary in place and resolve business intelligence inconsistencies by repairing existing business intelligence systems to adhere to that shared business vocabulary. In that sense, all business intelligence data stores and business intelligence tools need to:

1. Define and use common naming and common definitions for data across all business intelligence systems

2. Use common data integrity rules on common data items across all business intelligence systems

We also need to:

3. Identify (discover) the definitions of disparate data. This means discovering the metadata in these source systems and integrating the metadata. Given the disparate data in source systems and indeed the discrepancies across existing business intelligence systems, it would seem obvious that data integration software (extract, transform and load/ETL or enterprise information integration/EII software) could be used to integrate the disparate metadata describing disparate source data on these source systems and bring it into a repository for cross-reference to a shared business vocabulary. An example of this is Informatica (see Figure 2) who has used their ETL technology to integrate disparate data definitions and store that metadata in a repository.


Figure 2

The screen shot in Figure 2 shows how a metadata product can collect metadata and integrate it such that you have an “inventory” of disparate data items using Informatica. IBM with Rational Data Architect and Business Objects with Metadata Manager are other examples of vendors trying to address this problem.

4. Map the definitions of disparate data (cross-referencing data) in source system models to the definitions data items defined in the shared business vocabulary such that all data is correctly sourced and that we have full metadata lineage between disparate source data and common shared business vocabulary data used in business intelligence systems. Note that this is metadata to metadata mapping.

Figure 3 shows how disparate data definitions can be mapped from the source to the target shared business vocabulary definitions in IBM Rational Data Architect. By connecting to sources and relating each source to a shared business vocabulary, it becomes possible to understand what all the disparate data definitions for a data item or a master data entity mean.


Figure 3

5. Once these mappings are understood, the quality of disparate data content should be assessed.

6. At this point, it becomes possible to generate or define cleanup and translation rules for disparate data to get it into a common high quality state and then integrate that disparate data to create a common trusted set of data for a business intelligence system or indeed any other applications that need it.

While all these steps seem somewhat formal, they nevertheless make it possible to resolve inconsistencies in business intelligence systems and between source operational systems and business intelligence systems so that we reach a point where business intelligence is trusted and clearly understood. At this point, it becomes “safe” to then go to the next step – which is to look at integrating business intelligence into business operations.

To conclude, a shared business vocabulary is the foundation stone to consistency and integrated business intelligence. A model-driven data integration strategy is the way to achieve consistent common metadata. This is done by defining the entities and attributes in the shared business vocabulary and then discovering all data definitions in source systems that refer to each data item in the shared business vocabulary (e.g., all the data names and definitions for customer, product, store, etc. This discovery can be done using data integration software to integrate the disparate metadata in order to create an inventory of disparate data names and definitions for all data items. Note that this inventory of data definitions could be stored in a physical repository (created by ETL) or in a virtual repository (presented via EII). From here, we map disparate source system metadata (definitions describing source system data) from source system models to the shared business vocabulary (metadata describing business intelligence system data) so that we know precisely how the lineage works. Having done this, we can then define transforms to get the data from an unofficial state into a high quality commonly understood state to populate a data warehouse, for example. Note also that all business intelligence tool business views would then use the shared business vocabulary definitions to access data in the business intelligence system.


Figure 4

Once data is consistently defined in business intelligence systems, we are ready to integrate business intelligence into the enterprise to deliver just-in-time business intelligence into operational business processes to guide operations.

Note, however, that once we know how disparate data maps to commonly defined shared business vocabulary data, we can use that knowledge way beyond business intelligence systems. For example, if we want to use SBV commonly defined data entities in an enterprise information integration view, we can use the metadata we have created to generate EII virtual SBV views and to generate mappings in a specific enterprise information integration tool as shown in Figure 5 in BEA System’s AquaLogic Data Services EII tool.


Figure 5

In fact, we can go much further that this and generate mappings to map disparate application data to common SVB definitions in enterprise service bus/message broker software, master data management software and much more.

 

 


Figure 6

Recent articles by Mike Ferguson

 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!