If one looks at the past civilizations and tries to determine the key to their success, one characteristic seems to surface with almost all of them. They all had outstanding infrastructures. As might be expected, the failed civilizations typically had inadequate infrastructures.

An enterprise infrastructure is to business intelligence (BI) applications what a transportation infrastructure is to automobile owners. In order to safely and comfortably travel with an automobile, there must be a suitable physical (could also be called technical) infrastructure, such as roads, bridges, traffic lights and traffic signs, as well as non-physical (could be called non-technical) infrastructure, such as standardized traffic rules and their uniform interpretation. For example, without the universal interpretation of the rule that “green means go and red means stop,” traffic lights would be useless and could even be hazardous.

An enterprise infrastructure for operational business intelligence applications consists of two major components:

• Technical infrastructure, such as hardware, middleware and database management system (DBMS)
  
  
• Non-technical infrastructure, such as standards, metadata, business rules and policies

A word of warning – don’t expect to buy an operational business intelligence turnkey solution. An operational BI environment evolves over time with its infrastructure!

Hardware
For adequate report and query performance, it is imperative to have sufficient “horsepower” with the hardware platform. Scalability is the key. The biggest problem (albeit a desirable one) an organization has to face involves growth. What will the organization do if it is wildly successful? In order to be able to handle success graciously, most organizations implementing BI applications have to consider at least four factors in hardware platform selection:

1. New hardware platforms have to fit in the existing hardware configuration.
   
   
   • Is new hardware needed? What will it be? How much will it cost?
     
     
   • How well will the new hardware integrate with existing platforms?
     
     
   • How will the new hardware scale? In conjunction with the existing hardware? Standalone?
     
     
   • Will more staff be required to maintain the new hardware?
     
     
2. Performance of the new hardware together with the existing DBMS has to be outstanding. The DBMS on the selected hardware platform must perform well as database access and usage grows. It cannot be overemphasized that scalability is one of the major issues to be addressed with any DBMS. One maxim to remember is that a database always grows – it never shrinks.
   
   
3. Interoperability of hardware platforms: platform selections will be restricted by the need for interoperability between various hardware platforms (if required).
   
   
4. Cost and return on investment (ROI) for the previous three qualifiers are controlling factors.
   

Hardware Platform Requirements
The hardware must have sufficient power to handle complex access and analysis requirements against large volumes of data. It has to support not only predefined, simple queries on summary data, but also ad hoc complex queries and reports on detailed data. It must also be scalable because rapid changes will occur in:

• Data volume
  
  
• Updating frequencies
  
  
• Data access patterns
  
  
• Number of reports and queries
  
  
• Number of people accessing the BI target databases
  
  
• Number of tools running against the BI target databases
  
  
• Number of operational systems feeding the BI target databases
  
  

It is appropriate to think of an operational business intelligence environment in terms of a three-tier computing architecture (see Figure 1).

Figure 1: Three-tier Computing Architecture

Middleware
Middleware refers to runtime system software, which is layered between the application programs and the operating system. It acts as a bridge to integrate application programs and other software components in an environment with multiple network nodes, several operating systems and many software products. Watch out for products that offer technical elegance that is not needed. The price paid for this elegance is either more staff training or more resources – and most probably both.

Middleware is needed to run client/server implementations and other complex networked environments in a distributed computing environment. It must be determined if the organization has the necessary middleware to retrieve the source data from heterogeneous platforms and transfer it to the BI application environment.

Most of the middleware falls into two major categories: distributed logic middleware and data management middleware, as shown in Table 1.

Table 1: Distributed Logic Middleware vs. Data Management Middleware

Database Management System (DBMS)
Items to consider when making decisions about the DBMS include:

• What database management systems are already in place?
  
  
• Will it be necessary to buy a new DBMS? What will it cost? Is there enough staff with the skills necessary to support the new DBMS environment? If not, what will it take to acquire the skilled staff?
  
  
• Will more database administrators need to be hired?
  
  
• Will the new DBMS be compatible with current operating system(s)?
  
  
• What software tools can run with it?
  
  

DBMS Function Selection Criteria
The following basic functions are some of the important and necessary attributes of a DBMS for handling the workload of a large business intelligence target database or very large database (VLDB):

• Degree of parallelism in handling queries and data loads
  
  
• Intelligence in handling dimensional data models and optimizers
  
  
• Database scalability
  
  
• Internet integration
  
  
• Availability of advanced index schemes
  
  
• Replication on heterogeneous platforms
  
  
• Unattended operations
  
  

DBMS for the Operational BI Environment
A DBMS is a sophisticated piece of software and consists of a number of features that need to be evaluated. Features to look for in the DBMS for business intelligence applications include:

Network support. The network support provided by the DBMS should be compatible with the organization’s data communications standards.

Dimensional capability. For better performance, it is necessary to have dimensional capability in the form of seamless support for fast and easy loading and maintenance of pre-compiled summaries.

Adequate state-of-the-art triggers and stored procedures. It is important that procedures can be used as “event alerts,” which trigger an action in response to a given set of circumstances (e.g., cash flow below a certain level).

Administrative support features. These features should provide for maintenance of consistent historical data; support for archiving (for example, drop the oldest week when a new week is added); controls for implementing resource limits to display a warning when a query that consumes excessive resources is about to be terminated; workload tracking and tuning mechanisms; and careful monitoring of activity and resource utilization.

Location transparency across the network. This feature must allow the access and analysis tools to retrieve data from multiple BI target databases from a single workstation.

Future usage explosion. Future usage must be supported by effective caching and sharing of data to minimize input/output (I/O) bottlenecks, by the ability to effectively manage task switching of many concurrently running queries and by compatibility with multiple processors.

Scalability. Don’t jump to implement a VLDB unless the vendor’s VLDB features have proven themselves. The DBMS must have the capability to support:

• Advanced functions for sorting and indexing
  
  
• Fault tolerance for uninterrupted processing
  
  
• Uninterrupted maintenance operations, such as unload, backup and restore
  
  
• Taking checkpoints, recovery and rapid restart of interrupted operations

Query performance optimization. The query performance optimization should address CPU-intensive aspects of query processing such as joins, sorting and grouping.

Load process and performance. The chosen DBMS must address:

• Data obtained directly from a variety of feeds including disk files, network feeds, mainframe channel connections and magnetic tapes.
  
  
• Complete data loading and preparation, including format conversion, integrity enforcement, and indexing.
  

Security system. The security system must support unique passwords, password protection and authorization constraints necessary for specific persons and for specific tables of the database. The system administrator should provide restricted access to the views and virtual tables.

Data repository. The data repository should feed into a metadata repository, and the database objects should be linked to all data objects described in the enterprise logical data model.

Technical Infrastructure Activities
Review the existing platform in terms of hardware, middleware, DBMS and tools. It is important to evaluate the interdependence of the tools for their various purposes, such as the interdependence between a multidimensional reporting tool and an ad hoc querying tool. In addition, review the existing network architecture. One of the biggest bottlenecks today, especially in organizations with decentralized applications, is the lack of bandwidth coupled with a limited capacity for network growth.

After assessing the existing platforms, identify which types of new hardware, software or networking component must be acquired. If the existing hardware platform appears to be sufficient, be sure to determine that it will be able to provide the productivity and performance that is expected from it. Engage business representatives and stakeholders in the decision-making process by including them in peer reviews during the selection process.

Compile all findings about the existing platform into a report. Explain the strengths and weaknesses of the current hardware, middleware, DBMS and tools. Provide a list of missing technical infrastructure components necessary to meet the project requirements.

A technical infrastructure assessment report should itemize the scalability and limitations of the hardware, middleware, DBMS and tool platform. It should cover the following:

• Servers
  
  
• Client workstations
  
  
• Operating systems
  
  
• Middleware
  
  
• Custom interfaces
  
  
• Network components and bandwidth
  
  
• DBMS functionality and utilities (backup and recovery, performance monitoring)
  
  
• Development tools such as computer aided software engineering (CASE) and extract/transform/load (ETL) tools, access and analysis tools such as online analytical processing (OLAP) and report writers
  
  
• Metadata repository
  

Include a gap analysis section and provide recommendations for upgrading the platform. Incorporate the product evaluation and selection results, listing the weighted requirements and the product features that were evaluated.

Installation of Selected Products
If new products were identified, write a request for proposal (RFP) or a request for information (RFI) and send it to the vendors on the short list. After selecting a product, order, install and test it.

Tips for Success
Remember that one of the main features required to support an operational business intelligence environment is scalability. Pay attention to scalability; avoid any components that scale poorly. Scalability is one of the most important factors to be considered. Therefore, monitor the rapid changes in:

• Data volume
  
  
• Load frequencies
  
  
• Data access patterns
  
  
• Number of reports and queries
  
  
• Number of tools (Note: Use tools whenever possible instead of writing your own custom code. Understand the types of analysis the business people need to perform so that you can choose the appropriate tool set.)
  
  
• Number of people and types of users accessing the BI target databases
  
  
• Number of operational systems feeding the BI target databases
  

The necessary and important functions of the selected DBMS for the operational business intelligence environment are: the degree of parallelism in handling queries and data loads, intelligence in handling dimensional database designs (optimizers), database scalability, Internet integration, availability of advanced index schemes replication on heterogeneous platforms and unattended operations.

In order to provide adequate performance in a growing operational business intelligence environment, it is mandatory to assess the hardware, middleware, DBMS and tools from time to time. If this is not performed, it is conceivable that performance could degrade to such an extent that the operational BI environment will become unusable.

It is also necessary to stay current with the existing technology. Technology advances occur every few months. Not staying current and not taking advantage of new and improved features can turn the operational BI environment into an extinct dinosaur in a very short time.

Source: Moss, Larissa and Atre, Shaku. Business Intelligence Roadmap – The Complete Project Lifecycle for Decision-Support Applications, Addison Wesley Professional, 2003.)

Copyright Shaku Atre 2006. All rights reserved.

Recent articles by Shaku Atre

• What Is So Different About Business Intelligence Today?
• Data Analysis: A Pivotal Activity for Operational Business Intelligence, Part 2
• Data Analysis: A Pivotal Activity for Operational Business Intelligence, Part 1
• Two Main Pillars of Operational Business Intelligence, Part 2