Some analysts use the term “lights out” to describe these integration scenarios, but lights out doesn’t necessarily imply hands off. Once companies set up the necessary business to business (B2B) and application to application (A2A) links using integration software, they look for ways to easily and accurately monitor these automated processes.

Integration platforms incorporate business activity monitoring (BAM) tools to monitor transaction-processing activity. These tools generally maintain information in textual event logs. Administrators can review these logs to ensure that service levels are being met; and they can generate reports about transaction volume, throughput and exceptions. But in the age of e-business, textual event logs are no longer sufficient. Organizations are ramping up their online trading activities, and they are demanding analytic reporting capabilities to analyze the huge amount of information that’s being exchanged. Instead of a simple command-line interface, they favor web-based dashboards that present domain-specific information about system activity, business activity and trading partner activity.

Advanced integration platforms include these dashboards, along with embedded business intelligence (BI) capabilities to help administrators quickly troubleshoot problems, detect anomalies and conduct capacity planning exercises. In addition to auditing trading partner exchanges and sending alerts that trigger process adjustments, these BI solutions can alert individuals to changes and exceptions that may require action, as well as provide aggregated insight for strategic planning.

Integration platforms with embedded BI tools are ideal for presenting and analyzing real-time transaction information. This article suggests what to look for in a complete integration platform and how to use BI technology in conjunction with BAM tools to achieve zero information latency and self-correcting processes.

Electronic Trading: The Rise of B2B

Let’s begin by examining the business and technology trends that are bringing these issues to the forefront of data center operations, most of which center around electronic trading and e-business.

Clearly, e-business has evolved to include much more than using the web as a storefront. Today’s e-business applications support electronic business models that enable companies to link their internal and external data processing systems very efficiently. E-business is more than just e-commerce. It involves business processes spanning the entire value chain: electronic purchasing and supply chain management, processing orders electronically, handling customer service and cooperating with business partners. Companies use e-business applications to shorten supply chains, streamline distribution processes, reduce inventory carrying costs and many other online activities. Typically, these organizations set up electronic linkages to synchronize activities with suppliers, minimize on-hand inventory, reduce distribution costs, and to bring buyers and sellers together through electronic means.

While there are costs associated with setting up and maintaining these electronic links, the savings far outweigh the costs – especially if you have comprehensive monitoring tools to manage and audit these processes. For example, if you are automatically restocking inventory as partners place orders for merchandise, you need to make sure that each restocking request is responded to in a certain period of time. If not, an administrator needs to be notified to take immediate action.

There are two aspects to this process. In the first case, administrators need to be apprised of how their transaction systems are running so they can ensure that service levels are being met. In the second instance, they must be able to query the log files and generate reports about transaction volume, throughput and exceptions.

Straight Through Processing: Who Turned Off the Lights?

The culmination of these e-business activities is often referred to as straight-through processing (STP) since they enable trading processes to be conducted electronically without requiring an administrator to re-key data or intervene in the workflow. STP technology began in the financial services industry within capital markets, extended to payment-based transactions, and now impacts many other industries including manufacturing, insurance, energy and banking.

In some cases, electronic trading activities entail complex life cycles and a labyrinth of carefully coordinated processes. In the brokerage industry, the goal is to minimize settlement risk by executing trades then handling settlement and clearing simultaneously. In other industries, the objective is to speed the movement of funds, streamline the flow of goods, or automate the exchange of information. While common business processes like issuing a claim or restocking a warehouse might take several days from initiation to settlement, STP generally implies same-day processing or faster, often taking just a few seconds.

Historically, STP solutions were devised to help financial firms achieve same-day settlement of equity transactions, particularly in response to the explosive growth of online trading. Today, the basic concepts of STP are being applied to a variety of business domains to expedite trading relationships, reduce operational risks and create an audit trail for regulatory compliance. When properly deployed, all parties benefit from shortened processing cycles, reduced settlement risk and lower operating costs.

EDI and its Derivatives

EDI refers to the transfer of structured data, by agreed upon message standards, from one computer system to another without human intervention. It includes a variety of standard structures that emulate documents, such as purchase orders to automate purchasing. EDI transmissions can use many methodologies including modem (asynchronous, and bisynchronous), HTTP, AS1, AS2 and others. As more and more trading partners use the Internet to exchange messages and documents, some of the traditional EDI transmission methods are being replaced by Internet protocols such as FTP, telnet and email. But regardless of the transmission method and protocol chosen, administrators need a way to keep an eye on e-business activity.

The Rise of Business Activity Monitoring

The goal of business activity monitoring is to provide real time information about the status and results of various operations, processes and transactions.

BAM applies integration and business intelligence (BI) technologies to automated processes to continually refine them based on feedback from operational events.
BAM software enables an enterprise to make better business decisions, quickly address problem areas and take full advantage of emerging opportunities.

By proactively identifying problems and sending alerts accompanied by relevant information to appropriate systems or individuals, BAM solutions enable organizations to quickly resolve issues before they become critical – or take advantage of opportunities while they are still hot. For example, an administrator might want to fire off a query to view pending orders or see a list POs that haven’t been acknowledged within the last two hours.

BAM solutions can audit any business process in real time and provide feedback and associated data to improve e-business processes. This is an essential aspect of online trading relationships. For example, a BAM system might automatically generate an alert when a trading partner does not acknowledge the receipt of a purchase order within a pre-defined period of time. These alerts can be set to notify designated individuals by email, phone calls or text messages.

Setting Up Trading Partner Relationships

In order to see how a BAM system might work in an e-business context, consider a typical midsize company with a few dozen trading partners. This company probably exchanges hundreds of types of transactions. Each transaction includes four to eight messages, creating a great deal of interactions that must be managed.

When setting up mechanisms for interacting with each trading partner, IT pros must first answer a variety of questions:

• What protocols do we use to communicate with this partner (e.g., AS2, FTP)?
  
  
• What document formats do we use (e.g., XML, CSV, EDI)?
  
  
• What document types (e.g., EDI 850, RosettaNet PIPs)?
  
  
• What process do we execute while sending or receiving information (e.g., priority handling, different types or levels of information, etc.)?
  
  
• How do these processes relate to existing services?

For example, a process might call a service hosted by Supplier A using FTP, comma-separated value (CSV) files, and basic user id/password security. Calling Supplier B’s service may require AS2, XML formats and digital certificates.

Conversely, a process might use the same AS2 protocol, XML messages and digital certificates to communicate with two different suppliers, but handle each differently – for example, if one partner has contracted a higher service-level agreement (SLA) than the other.

With one or two business partners, setting up these interactions isn’t too difficult. But as organizations add more partners, they need integration tools to automate the connections, along with BAM tools to control how they interact with them.

Most companies use some type of business-to-business (B2B) integration tool to create and manage these trading partner exchanges. At the very minimum, these tools include the following:

• An auditing facility that can be configured to record any stage of message processing.
  
  
• A correlation manager to identify the status of each message and expected acknowledgments.
  
  
• Profiling facilities that reveal variables about each trading partner, correlation information and other important B2B information such as transformations, process flows, access keys, error handler contacts and so forth.
  
  
• Some type of reporting facility for monitoring the exchanges and troubleshooting problems. These facilities range from simple event logs to full-blown BI environments.

Where BAM Meets BI

Most BAM systems maintain transaction information in rudimentary event logs. System administrators rely on this data to monitor activities and track down errors. But in the age of e-business, textual event logs are no longer sufficient. Organizations are ramping up their online trading activities, and they are demanding analytic reporting capabilities to analyze the huge amount of information that’s going back and forth.

That’s where business intelligence (BI) comes in.

Business intelligence refers to technologies, applications and practices for the collection, integration, analysis and presentation of business information. BI systems provide historical, current and predictive views of business operations, most often using data that has been gathered into a data warehouse or a data mart. Today’s BI systems often display information through dashboards via key performance indicators (KPIs) that measure the performance of critical business activities. These dashboards also support root cause analysis and generate alerts to warn of impending problems.

Although BAM systems usually use a dashboard to present data, these dashboards have some distinct characteristics not always found in BI tools. For example, management dashboards are refreshed at predetermined intervals by polling or querying databases. BAM systems push information to dashboards in real-time as messages are sent, received and processed. BAM systems are driven by business events – in this case, fed directly from integration software. These systems are dynamic, meaning the display is not dependent upon a user refreshing the data.¹

From Event Logs to BI Dashboards

What do you get from a dashboard that you won’t find in a simple event log? For starters, dashboards use an intuitive visual metaphor to track the flows inherent in the underlying business processes that they monitor. In this case, they enable system administrators to gauge trading partner activities by capturing and reporting designated data points. This might include the following:

• A visual presentation of performance measures
  
  
• The ability to identify and correct anomalies
  
  
• An ongoing measure of trading partner efficiency
  
  
• The ability to generate detailed reports on demand
  
  
• An easy way to audit trading partner exchanges
  
  
• A central console for sending alerts and triggering process adjustments
  
  
• Aggregated insight for strategic planning

In conjunction with the visual display of a dashboard, associated BI tools can help systems administrators analyze, query and display trading partner information in meaningful ways. The dashboard might include links to commonly used reports that summarize messages sent and received, correlation history, quality of service metrics, and other current information. This saves a lot of work since people can quickly analyze this information without having to create reports or construct a visual interface.

Example Reports and Displays

Most integration vendors have added some type of reporting system to their activity monitors. A complete BI environment adds the ability to query, report, analyze, deliver and display electronic trading data in any way you choose. This enables administrators to view, monitor and report on the associated integration processes. A complete solution can capture end-to-end transaction and workflow data across multiple applications and business units, summarizing and displaying trading partner metrics to help managers make informed decisions.

BI monitoring tools generally include useful standard reports right out of the box. The following figures from the iWay Activity Monitor are a good example.
 
Figure 1 reveals all open activity, including unacknowledged messages that haven’t been responded to in a certain amount of time.

Figure 1: A sample correlation management report from the iWay Activity Monitor.

If these reports are rendered as simple HTML displays, organizations can extend powerful administrative capabilities to anyone who needs them, without any additional client software, plug-ins, or report viewers. Figure 2 reveals a report of past messaging activity.

Figure 2: A report revealing the correlation history for trading partner activity.

This particular activity monitor generates performance metrics in real-time and publishes these metrics by the second, minute and hour. It’s built on a publish/subscribe model, so clients don’t have to poll the server and the server doesn’t need to repeatedly query or calculate this information. This process and the accompanying activity display are shown in Figure 3.

Figure 3: An alert-driven activity monitor automatically pushes performance metrics to clients.

Customizing the Environment

There is no “cookie-cutter approach” to establishing trading partner relationships. Thus, a good activity monitor should be customizable so it can be extended and personalized to fit individual needs. As shown in Figure 4 and succeeding diagrams, system administrators should be able to easily do the following:

• Create users and roles
  
  
• Define and customize reports and determine when/where to display them
  
  
• Define new dashboards or “workspaces”
  
  
• Define performance metrics that determine what the server should report on and which widgets to display
  
  
• Manage trading partner information

Figure 4: An activity-monitoring dashboard.

 
Figure 5: Editing widget properties to customize the environment.

Figure 6: Creating a custom role for the helpdesk.

Use Cases

For example, consider a logistics and package delivery company in the United Kingdom, which is using a BI activity monitor to audit business processes related to fulfillment processes. The environment captures bills of lading and transfers them into the company’s package-tracking systems, where workers can quickly produce reports from the data.

These shipping and logistics systems must work with the utmost precision in order to keep customer deliveries on schedule. The failure of any vital communication could delay an important shipment. Developers have programmed an activity monitor so it is aware of each message and awaits confirmation. Failure to receive confirmation within a reasonable amount of time raises an alarm so administrators can take immediate action. They can investigate the cause of the delay and push the problem toward immediate resolution.

A large retailer, which preferred not to be named for competitive reasons, depends on BAM/BI tools to manage order-processing activities and ensure new customer accounts are set up correctly. These activities entail communication with the systems handling customer records, inventory and billing information. This company’s integration environment processes low-level technical events such as messages passing from one application to another and transactions detected within a database log file. All events relating to an individual customer are correlated to spot anomalies, such as an order that has not been promptly provisioned or a customer record that lacks a valid billing address. If something is amiss, the activity monitor generates an alert to notify the technical operations staff.

Another great example concerns a chemicals company based in The Netherlands, which is in the process of replacing a trading partner system based on SAP and Microsoft BizTalk middleware. IT pros at this company established an electronic trading environment that includes other chemical companies on the supply side and customers/distributors on the demand side. Their system-based mappings include demand forecasts, purchase order updates, invoices, shipping requests, inventory levels and advanced ship notices. While their former BizTalk interface revealed standard information about these messages, the new activity monitor leverages embedded BI tools to provide highly visual and detailed reports, so administrators can more easily resolve problems.

What leads companies to adopt more robust business activity monitoring tools? The turning point for this company was a need to upgrade the BizTalk environment, along with the accompanying custom-developed BAM system. IT pros there realized they could deploy a turnkey solution that could grow with the business, replacing custom code with standard components and an off the shelf BAM environment. This environment now manages communications across HTTP, HTTPs, AS2, RNIF/CITX, web services and SMTP.

Conclusion

Companies and government agencies are approaching the goals of zero information latency and self-correcting processes. As they use automated systems to realize this vision, they depend on BAM solutions to fine-tune the exchange of information, monitoring everything from package deliveries to payments. A complete BAM solution results in more accurate resource allocation and more efficient trading partner relationships.

As organizations move to lights-out operations and straight-through processing, they need sophisticated integration capabilities bolstered by mature tools for business activity monitoring, reporting and analytics. An integration platform that includes BI technology is ideal not only for setting up and managing trading partner relationships, but for generating alerts, creating dashboards and monitoring all aspects of B2B activity.

Reference:

1. “Dashboards by Example,” Volume 1, http://www.enterprise-dashboard.com/.