We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Rapid Delivery Technologies for Enterprise Architecture: Enterprise Application Integration (EAI) Concepts

Originally published March 8, 2007

Previous articles have introduced enterprise architecture (EA) rapid delivery methods and described past EA projects for government and banking. This month, I will start to cover some of the technologies used for rapid delivery into production. I will start with some fundamental principles relating to enterprise application integration (EAI). I will then cover Web services, service-oriented architecture (SOA) and also discuss business process management (BPM) languages.

Technologies for Enterprise Integration
Very effective technologies are now becoming available for enterprise integration. These achieve technology integration, based on the use of XML (extensible markup language), enterprise application integration and Web services. I will briefly introduce XML here.

A prerequisite for the effective use of XML is data modeling: to define the metadata that XML requires. Many data modeling tools automatically generate XML definition files from defined metadata.

XYZ Corporation is a hypothetical organization that is a supplier to two other organizations, ABC Inc and DEF Enterprises, who buy products from XYZ. In doing business with its customer ABC Inc, XYZ carries out the following business activities:

  • The XYZ sales department accepts sales orders from its customer ABC to be processed by the XYZ order entry department. This is carried out by the order entry management activity.
  • Before an order can be processed, the XYZ credit control department first checks the available credit for its client ABC based on the current values of client account balance and client credit limit, and the total amount of the requested order. This activity is called credit control management to authorize order entry processing.
  • After the order entry department has fully processed the order, the accounts receivable section in the XYZ finance department uses the customer invoice management activity to issue an invoice. This increases the debtor account balance of its debtor ABC by the total amount of the order.
  • The new debtor account balance is used to update the redundant versions of the same amount due to XYZ: the customer account balance for customer ABC in the sales and order entry departments, and the client account balance for client ABC for the credit control department. This activity is required for data maintenance synchronization of these redundant versions of account balance. This is the responsibility of the accounts receivable management activity.

Because of the different terminology and identification used by each area of XYZ, it is not always clear they are dealing with the same organization: ABC. For example, customers of XYZ are identified in the sales and order entry departments by customer number: ABC has a unique customer number. Clients are identified in the credit control department by client ID, with a unique client ID for ABC. Debtors are identified in the finance department by debtor code, again with a unique debtor code for ABC. But due to the lack of business integration, ABC Inc now appears to XYZ as three quite separate organizations: as a customer, as a client and as a debtor. Let us examine how XML handles this situation.

Basic XML Concepts
XML indicates the context of relevant data by surrounding that data with “tags” that define its meaning. For example, sales orders from ABC appear in XML as shown next.

The start and end tags clearly indicate the terminology that is used in each department. We can see from these XML data fragments that each department is dealing with the same enterprise: ABC.

Business Documents
Let us now discuss the relevant activities and the corresponding business documents that are exchanged between customer ABC and its supplier XYZ.

  • The ABC purchasing activity produces purchase orders that are sent to XYZ as its supplier. These purchase orders can be sent by mail or fax.
  • When received by the XYZ order entry department, they are manually entered as sales orders by the XYZ order entry management activity.
  • On sales order acceptance after credit control management processing by the XYZ credit control department, XYZ sends an order acknowledgement back to ABC, followed later by a delivery advice or advance shipping notice (ASN). These documents are received by the ABC purchase order acceptance activity.
  • Finally the accounts receivable section in the XYZ finance department sends a customer invoice to its debtor: ABC. It is received by the ABC accounts payable activity in their finance department as a supplier invoice – from its supplier XYZ.
  • All these documents are exchanged between the two companies: the purchase order originating the order at ABC; the order acknowledgement of receipt by XYZ, sent back to ABC; the delivery advice or advance shipping notice; and later the invoice that is sent by XYZ to ABC. Each of these documents must be manually reentered by XYZ or by ABC staff into relevant XYZ or ABC processing systems.

Electronic XML Documents
An example of the purchase order that is exchanged between both companies is shown in Figure 1. This purchase order (PO) is issued by “Smith and Co” – a subsidiary of ABC. It is a purchase order expressed in XML. We will use this PO to discuss some of the basic concepts of XML, documented as a summary in Box 1. (Introductory papers and white papers on XML can be found at the World Wide Web Consortium – W3C, and the Organization for Advancement of Structured Information StandardsOASIS.)

Figure 1: Example of an XML purchase order. (© 2001 Software AG. Reprinted with permission.)

The Need for XML Transformation
Consider now that Smith and Co is no longer a subsidiary of ABC Inc, but is a subsidiary of DEF Enterprises. The purchase order instead would use the XML PO format that is utilized by DEF. This is shown in Figure 2: side-by-side with the same PO in the XML format that is used by ABC. This purchase order is OrderNo:1234.

We see that the DEF PO format on the left shows the order number as an attribute of <<em>PurchaseOrder OrderNo="1234">. On the right, it is shown as an XML element <<em>OrdNo>1234.

The Buyer Number identification and name on the left for DEF are shown as attributes of Buyer: <<em>Buyer BuyerNo="AB24567" Name="Smith and Co">. On the right for ABC, each is an XML element: <<em>Reference>AB24567<<em>/Reference> <<em>Name>Smith and Co.

On the right, the address is <<em>Street>123 High StEpping Forest<<em>/Town> <<em>PostCode>E15 2HQ<<em>/PostCode>. On the left, the address uses quite different element names: <<em>Address1>123 High St<<em>/Address1> <<em>Address2>Epping Forest<<em>/Address2> <<em>Zip>E15 2HQ<<em>/Zip>.

Figure 2: A purchase order in two different XML formats. (© 2001 Software AG. Reprinted with permission.)

Finally, we can see that XML attributes for <<em>Tax> are used by ABC on the right, as: <<em>Tax Type="VAT" Percent="17.5>. In contrast, the format on the left uses child elements of <<em>Tax>. These are: <<em>TaxType>VAT<<em>/TaxType> and <<em>TaxPercent>17.5<<em>/TaxPercent>, surrounded by <<em>Tax>  … <<em>/Tax> as their parent element.

From Figure 2, we see that these purchase order formats in XML are quite different from each other, yet both contain exactly the same PO data content. Both ABC and DEF are customers of XYZ, which likely uses a different XML format for its sales orders. As noted in the figure, different organizations can all define the same data in different ways. Each of the customers of XYZ may have quite different PO formats.

So what should XYZ do? Should it require every customer to use a standard format, such as the XYZ sales order format? This would be very convenient for XYZ, but it would not be so convenient for its customers. Rather than change their PO formats so they could buy from XYZ, it is easier for them not to buy. Instead, they can buy from a supplier other than XYZ. If XYZ mandated that only its XML sales format be used, it would quite likely lose many customers and their orders.

Because each customer most likely has its own PO format in XML, to attract their continued business, XYZ must be able to accept every different format. It therefore must be able to transform each format into its own sales order XML format. This is one of the advantages of XML: it is very easy to do data transformation using XML.

Extensible style language transformation (XSLT) is used to transform from one XML format to another. (Extensible Style Language Transformation (XSLT) and Extensible Style Language (XSL) specifications and white papers are available fromW3Cand fromOASIS. The XSLT language specifies in XML the transformation that is to be carried out, processed by an XSLT processor – called an XSLT transformation engine, as shown in Figure 3.

Each document to be exchanged between XYZ and ABC previously has had to be printed and then mailed – or faxed – as we discussed previously. It was then manually reentered by the receiving company into its relevant system for processing.

Figure 3: Example of the XSLT transformation engine used by BizTalk Data Mapper. (© 2004 Microsoft Corporation. Reprinted with permission.)

However, this manual reentry step changes with XML. Each document is intercepted before printing and is instead converted to an XML electronic document. This is then transmitted across the Internet, or within the corporate intranet, as text in XML.

Using XML, the manual reentry step is bypassed. These electronic documents are sent automatically as XML messages between the relevant XYZ and ABC systems.

XML messaging allows these enterprise applications to be integrated. Not surprisingly, this is called enterprise application integration. EAI is used across enterprise boundaries between XYZ and ABC as inter-enterprise application integration. Used within an enterprise, such as in XYZ, it is called intra-enterprise application integration.

A transformation front end is added to each system for EAI between the various systems. In normal processing workflow, these systems are: ABC Procurement; XYZ Order Entry; XYZ Order Acknowledgement; XYZ Advance Shipping Notice; ABC PO Acceptance; ABC Advance Shipping Notice; XYZ Customer Invoicing; and ABC Accounts Payable. An example of an XSLT engine used for transformation of data formats is illustrated in Figure 3 for Microsoft BizTalk.

B2B Cost Effectiveness Business Drivers
Some of the reasons for rapid acceptance of e-business are due to its cost effectiveness, with dramatic cost savings. The savings from e-business can be best understood by looking at the associated costs for purchasing and invoicing in enterprises.

According to Gartner, a typical purchase order, with approvals, costs approximately $100 to $150. This transaction processing cost is static. There is no difference whether the item purchased is a $5,000 computer, a $5 million piece of equipment or a $5 box of pens. In each of these cases, the $100 to $150 transaction processing cost remains the same. Gartner has found that 75 % of items in a typical Purchase Order or Invoice cost less than $1,000. So this processing cost makes up a significant proportion of the cost of each item.

The transaction processing cost applies regardless of the currency that is involved. For example, the same $100 - $150 transaction cost applies whether expressed as $USD in the U.S., $CND in Canada, $AUD in Australia or as £UK in England. The cost has nothing to do with currency exchange rates; rather, it depends on the cost of living in each country. This is due to the manual processing that is required to key the details of all items in a purchase order, to correct any data entry or other errors, to authorize the purchase order and to mail or fax it to a supplier for fulfillment. All of these manual steps have been required historically.

Similar costs are incurred when the purchase order is received as a sales order by the supplier. As we discussed earlier, details of the sales order must be manually entered into an order entry system for processing, any data entry or other errors must be corrected and each ordered item must be checked for available inventory and pricing.

On acceptance of the sales order, an advance shipping notice is then printed to be mailed or faxed back to the customer. (An ASN typically provides confirmation of acceptance by a supplier of a purchase order issued by a customer, and advises the price and planned date for delivery of the order to the customer.) An invoice is also printed to be mailed or faxed to the Customer. We now see why this manual data entry and processing typically costs $100 to $150 for purchase orders. A similar cost is also incurred with manual reentry and processing of sales orders and invoices.

We discussed earlier that manual data entry and correction steps can be bypassed by transmitting electronic XML purchase orders and invoices between customers and suppliers across the Internet. These XML documents can be transformed for automatic entry into order entry and other relevant systems without the need for manual reentry and processing. This has a dramatic effect on the costs that are incurred.

Using EAI as discussed in this article, this cost drops to $8 to $10 for transaction processing. These cost savings are enjoyed by both of the enterprises in a buyer-supplier trading relationship.

Using EAI, delays previously associated with manual data entry – for the correction of errors with related manual processing, followed by mailing or faxing – all disappear with electronic XML document transmission. The previous manual delays reduce to minutes, even seconds. EAI enables processing to be completed in near-real time.

Furthermore, an Aberdeen Group survey has indicated that the complete manual purchase requisition cycle lasts on average 7.3 days while this data entry, correction and related manual processing is completed. However, an automated buy-side e-commerce system using electronic data interchange (EDI) messaging reduces this time typically to 2 days, but the manual cost for reentry of transactions is avoided with EDI.

The Growth of Trading Communities
Let us examine further the impact that the Internet is having on business. With the easy interconnection offered by Internet technologies, any enterprise can now communicate instantly with any other enterprise anywhere in the world – as buyers and suppliers for B2B e-business.

The major problems associated with e-business are connecting everyone together and finding buyers and suppliers. The Internet offers the ability to easily interconnect buyers and suppliers in real time, and worldwide.

Connecting Enterprises Together
Trading communities (also called trading networks) have sprung up over the last few years to take advantage of the dramatic cost savings discussed earlier that are available from e-business. These enable enterprises to do business with other enterprises anywhere in the world, using a variety of messaging formats and protocols. These formats include RosettaNet, BizTalk, EDI, ebXML and others as illustrated in Figure 4. We will discuss these messaging formats later in this article.

Trading communities utilize the technologies of enterprise application integration and XML. A trading community can be horizontal across many industries, or vertical within a single industry. An example of a vertical trading community is Covisint, established in the U.S. between automobile manufacturers GM, Ford, Daimler/Chrysler, Nissan and more than 60,000 suppliers, with over $US240 billion in purchases per year.

Figure 4: Trading communities accept a variety of XML message envelope formats, automatically converting from one to the other. (© 2002 CommerceOne. Reprinted with permission.)

We discussed earlier – with XYZ Corporation, ABC Inc and DEF Enterprises – that organizations use different terminology and message structures as they do e-business together. We saw that there is a need for transformation between different terminologies (as XML metadata tags) and different XML message formats. Enterprise application integration is an important enabler for this transformation. There are several ways that enterprises can interconnect across the Internet, as discussed next.

Figure 5 shows that the simplest way that two enterprises can connect for e-business is via a Web browser. In this example, if Company 1 places an order with Company 2, it has to print out this order directly from the Web form used to order products online from Company 2. It must then manually reenter that online order as a purchase order into the back-end procurement system of Company 1. With this manual data entry, Company 1 does not benefit from any of the cost and time savings that we discussed earlier.

Figure 5: Using a Web Browser, Company 1 can connect easily to Company 2 and so purchase goods and services. (© 2002 CommerceOne. Reprinted with permission.)

Similarly, Company 2 may also have to print out all sales orders received online, such as from Company 1. If it does not have its online order acceptance system integrated with its back-end order entry system, it must also manually reenter those sales orders into the order entry system. With this manual data entry, Company 2 also does not benefit from any of the cost and time savings we saw earlier.

While this browser-based approach is inexpensive to implement, it does not result in the dramatic cost and time savings that can be achieved from bypassing manual reentry, by using electronic order entry through e-business.

Figure 6: Company 1 and Company 2 both use XML electronic purchase orders directly from or to back-end systems. (© 2002 CommerceOne. Reprinted with permission.)

In contrast, another approach is shown in Figure 6. This enables Company 1 (ABC, for example) to authorize a purchase order in its back-end supplier procurement system. This PO is not printed out and mailed or faxed to Company 2 (XYZ, for example) as has traditionally been done. The PO is transformed instead into an XML purchase order, which is transmitted electronically over the Internet to Company 2 (XYZ).

On receipt, the XYZ order acceptance system receives the PO sent by ABC now as a sales order. This sales order is transferred electronically – without requiring manual reentry as in Figure 5 – into its integrated back-end sales order entry system. Both of the companies in Figure 6 are now engaged in e-business: their online e-commerce systems are now closely integrated with their back-end processing systems.

The approach in Figure 6 bypasses the cost and delays previously associated with manual reentry. It enables dramatic cost savings to be achieved as discussed earlier. A typical static purchase order transaction processing cost of $100 to $150 that occurs with manual reentry is now reduced typically to $8 to $10 (for both Buyer ABC and Supplier XYZ), by using an electronic XML PO as illustrated.

Figure 7: There are many possible XML formats for document interchange between organizations (© 2002 CommerceOne. Reprinted with permission.)

As we discussed earlier, Company 1 and Company 2 (ABC and XYZ) must agree on the format of the ABC Purchase Order – received by XYZ as a sales order. We will call this agreed format: PO Type 1 – as shown in Figure 7. Both ABC and XYZ will need to transform between this agreed PO Type 1 format and its own internal enterprise format, based on each company’s own defined enterprise metadata.

In dealing with Company 1 and Company 3, Figure 7 shows that another format may also be needed: PO Type 2. Similarly, other PO formats may need to be agreed between each buyer and supplier pair. But Company 1 may have 1,000 different suppliers – each with a different PO format. And each of those may also have 1,000 other companies that they do business with – again, each with a different PO format. These companies, doing business with many other companies, result in 1 million separate transformation formats.

It is clear that this approach of many-to-many transformation pairs does not scale.

XML alone is not a complete answer. We discussed that the example shown in Figure 2 uses exactly the same data but with two different XML PO formats. These are only two of the million PO format pairs discussed in relation to Figure 7.

To address this problem of many-to-many transformations, common document formats are being defined in XML for each industry (see Industry Markup Vocabularies section), as illustrated in Figure 8. This shows a standard purchase order format, specified to address typical PO data as used in many industries. Because of the extensibility of XML, industry extensions can be defined easily to tailor a standard PO format for each industry, addressing their particular requirements.

Figure 8 shows the addition of travel industry PO extensions – and also auto industry PO extensions – to address the specific needs of companies in those industries. These PO formats for each industry can be used as a PO standard by all companies that do business with each other in that industry.

In effect, the many-to-many transformation problem that we discussed earlier has been resolved to a many-to-one transformation: from each company’s enterprise metadata to the relevant industry standard. Similarly, on receipt of this standard PO, each receiving company can do the corresponding one-to-many transformation: converting the received PO to each company’s own sales order format based on its own enterprise metadata. (This comment on many-to many transformations has no correspondence to the concept of many-to many associations that are used in data modeling.)

Figure 8: A common XML purchase order format can easily be extended using XML to address different industry requirements. (© 2002 CommerceOne. Reprinted with permission.)

This solution of XML format transformation is usually based on a defined industry markup language that specifies common metadata and XML document formats. (There are still integration problems encountered by cross-industry enterprises, such as government departments or diversified companies that may not have agreed common document formats.) These are used for business communication and data interchange between companies doing business together in an industry. Each company now only needs to convert to and from the agreed industry-standard XML format and its own enterprise metadata. This principle (of many-to-one and also one-to-many document transformation) is the basis of standard XML format transformation for industry markup languages in most industries. (To find out the industry markup languages that have been defined in your industry, visit the OASIS Web site. Click on the “XML Catalog” button in the left navigation frame. Scroll down the XML Catalog industry list displayed alphabetically in the right frame to locate your industry and follow the links.)

We now need to consider the problem of document delivery. In the real-world, a purchase order is printed by a buyer, then inserted in an envelope and mailed to a supplier as shown in Figure 9. On receipt, the supplier opens the envelope and delivers it to its order entry section for data entry and processing as a sales order. Mail delivery is fairly reliable: the mail is almost always delivered correctly. However it is slow: it can take days, sometimes even weeks for delivery of a mailed envelope.

Figure 9: Real-world document delivery. (© 2002 CommerceOne. Reprinted with permission.)

If fast, guaranteed delivery is required, a courier service is often used. This service is offered by the postal service in many countries, as well as by shipping companies such as FedEx, UPS or DHL.

Figure 10: Electronic document delivery. (© 2002 CommerceOne. Reprinted with permission.)

Contrast Figure 9 now with Figure 10, which shows electronic document delivery. The XML document can be attached to an e-mail message and transmitted electronically. But e-mail messages can be lost, or the same message may be delivered multiple times. To avoid problems of lost messages – or of multiple e-mail copies delivered from the one message – a more reliable method of electronic document delivery is needed.

UPS, DHL, FedEx and the Post Office use particular document formats for the envelopes that they use. If electronic documents are to be transmitted between computer programs, common formats are also needed for electronic envelopes.

Trading communities – such as CommerceOne and Ariba, and also message broker middleware products for a similar trading function – have emerged to address problems of reliable electronic document delivery and common envelope formats in XML. These transform documents between standard XML message envelope formats automatically.

Using this approach, an XML message sent by one company, which uses one standard XML message envelope, is transformed by a trading community or message broker to the specific envelope message format used by another company that it does business with. For example, CommerceOne automatically transforms message envelope formats to and from its own metadata language: xCBL (XML common business language).

In this way, messages that use standard XML formats such as RosettaNet (an XML format used by the IT and electronic components industries), EDI format, ebXML (EDI in XML), BizTalk (from Microsoft), OAG (Open Application Group) or OBI (Open Buyers Interchange) format are all automatically converted by Trading Communities or Message Brokers from one XML message format to any of the other XML formats. As well as these benefits of support for XML message envelope formats with automatic XML document translation, they also provide a reliable electronic “postal system” that guarantees delivery, with no lost or repeated messages.

Finding Buyers and Suppliers
A further consideration is the location of buyers and suppliers. Using standard Internet search engines to locate trading partners on the Internet is not comprehensive enough.

To address the problem of locating buyers and suppliers on the Internet, most trading communities allow suppliers to “advertise” the products and services that they offer. Using XML, they register and display on the trading community the products and services that they want to sell. Buyers can then visit the trading community and use XML to search for the products and services that they want to buy.

Alternatively, buyers use XML to register their tenders, requests for information (RFI), requests for quote (RFQ) or requests for proposal (RFP) with the trading community. Suppliers then use the XML search facilities offered by the trading community to locate each tender, RFI, RFQ, or RFP issued by relevant buyers that they want to respond to.

Figure 11: CommerceOne has established a Global Trading Web (GTW) using large telcos across many countries. (© 2002 CommerceOne. Reprinted with permission.)

Many trading communities have emerged in recent years. Trading communities survive by attracting many buyers and suppliers to trade with each other. A trading community typically generates revenue by taking a small percentage of the value of each trading transaction. So its ability to match buyers and suppliers is vital in generating future revenue for the trading community.

To survive, the multitude of trading communities that exist today will likely consolidate into one or two large communities in each industry or community group. Size, geographic scope and industry visibility are key factors in the survival of trading communities.

Figure 11 illustrates the Global Trading Web (GTW), established worldwide by CommerceOne. Separate trading communities are hosted by large telecommunications authorities in many countries, such as Japan, Singapore, Germany, the UK, Latin America and elsewhere. The CommerceOne software and facilities are commonly used by all, so that a Buyer in one country can trade easily with a supplier in any other country, using the GTW to communicate seamlessly.

Buyers and suppliers – using this approach through trading communities – can only find and do business with other companies that also use the same trading community. Any companies that do not use the same trading community are excluded.

What is needed is an electronic “Internet Yellow Pages.” This is similar in concept to the Yellow Pages printed directories in most cities and countries throughout the world: listing businesses under different categories for products and services that they provide. We will consider solutions to this problem in this and later articles, when we look at Internet Yellow Pages facilities based on ebXML and XML Web services.

We will now discuss the concepts associated with XML messaging standards in more detail.

XML Messaging and Repository Standards
XML messaging requires standards for XML envelope formats, as well as standards for XML business documents within those envelopes. As we briefly discussed earlier, there are many XML message envelope standards. Some also include business document formats for particular industries. The major envelope format standards follow:

  • RosettaNet: Used by the information technology and electronic components industries, RosettaNet defines XML message envelope standards and business document standards. The RosettaNet Web site also includes a repository of RosettaNet document formats freely available as document type definition (DTD) files and XML schema definition (XSD) files. We will discuss RosettaNet later in this article.
  • BizTalk: Microsoft BizTalk Server is widely used in many industries. It defines an XML message envelope format for guaranteed message delivery. It does not specify business document standards, but includes a Repository of many XML document formats. These are also freely available as Document Type Definition (DTD) files and XML Schema Definition (XSD) files, defining common XML business documents for use with BizTalk. Further Information about BizTalk is available from http://www.biztalk.org/. We will discuss BizTalk in more detail in the paper on EAI Vendors and Products.
  • ebXML: Electronic business XML (ebXML) extends the decades of experience with EDI to XML. The Organization for Advancement of Structured Information Standards (OASIS) together with UN/CEFACT (the United Nations body for trade facilitation and electronic business) are the standards bodies involved in the ebXML specifications for use with EDI. Like BizTalk, ebXML is a message envelope format – but it also includes a registry as an “Internet Yellow Pages” directory, as discussed earlier. OASIS and also http://www.ebxml.org/ include repositories that hold definitions of common XML document formats available for free use by other organizations considering ebXML and EDI interchange. We will look at ebXML in more detail later in this paper.
  • UDDI: Universal description, discovery and integration (UDDI) defines an XML registry for an “Internet Yellow Pages” similar to the http://ebxml.org/ registry. It is used with Web services defined using simple object access protocol (SOAP) and Web Services Description Language (WSDL). While UDDI provides much of the same functionality of ebXML, its focus is much broader than for use with EDI. (Information on UDDI is available from OASIS and from http://www.uddi.org/. Information on SOAP and WSDL is at http://www.w3c.org/). We will discuss Web services, SOAP, WSDL and UDDI in detail in a later article.

Industry Markup Vocabularies
As we discussed briefly earlier, many industries have defined industry markup vocabularies for XML document interchange and metadata communication. These vocabularies define various business document formats used in each industry and so become industry standards. Examples of some industry markup vocabularies include:

  • Banking and Finance: A cooperative Banking and Finance industry effort has defined Industry Markup Languages among major financial institutions – and the service providers and information technology vendors – plus their customers in small business and consumer markets. Information on banking and finance industry markup languages is available from their Web sites: IFX – Interactive Financial eXchange), OFX – Open Financial eXchange) and FpML – Financial Products Markup Language.
  • Audit and Financial Reporting: Defined as a universal language for financial information, reporting and analysis, the eXtensible Business Reporting Language (XBRL) addresses the problem of a lack of financial reporting standards. It resolves the problem of ineffective communication to investors, with inefficient aggregation and analysis, and inefficient creation of financial statements. XBRL standards bodies exist in many countries, including XBRL USA, XBRL Australia, XBRL Canada, XBRL UK and others. (Information on XBRL is available from http://www.xbrl.org/. This includes the use of XBRL for financial statements, general ledger, journal entry reporting, credit reporting, risk reporting, regulatory filings, assurance schedules, tax filings and for “business” reporting.)
  • Healthcare:HL7 (Health Level 7) is a healthcare industrial markup language to define standards for exchange, management and integration of data that supports clinical patient care and the management, delivery and evaluation of healthcare services. HL7 standards are defined for: message interchange, software components, medical logic, and document and record architectures.

We will now discuss some of the concepts that are associated with the aforementioned major XML messaging standards in more detail.

RosettaNet is a complete specification for XML messaging. It defines message envelope formats, business document formats and processes for the information technology (IT) and electronic components (EC) industries. It was cooperatively defined from 1999 by several hundred companies in these industries, with some listed in Table 1.

RosettaNet is based on the concepts of partner interface processes (PIPs). A PIP is defined as follows:

A PIP depicts activities, decisions and interactions that fulfill a business transaction between two partners in the supply chain.

PIPs are fundamental to the operation and use of RosettaNet. Not only does RosettaNet define one of the most complete industry markup vocabularies, it also extends beyond IT and EC common interchange XML document formats and industry terminology to defined infrastructure for messaging. It has established a legal foundation for B2B e-business that also benefits other industries. Many middleware tools support RosettaNet.

Table 1: List of organizations that contributed to development of RosettaNet.

The relationship between the concepts of a PIP and EDI are shown in Table 2. While a PIP focuses on processes and thus is process-centric, electronic data interchange focuses on data in messages and is message-centric. A PIP involves real-time, dynamic processing while EDI addresses batch processing of large volume EDI files.

Table 2: Comparison between RosettaNet PIP and EDI.

EDI is used for approximately 10% of the B2B processes between large trading partners, while a PIP addresses 100% of the processing of IT and EC B2B processes. EDI has been in use for decades, based on expensive, leased lines used as value added networks (VAN). In contrast, RosettaNet is designed for inexpensive use of the Internet.

EDI is based on defined standards – X12, EDIFACT or JECALS – while a PIP is based on the use of XML. EDI is intended for regional use by large trading partners, while a PIP can be used globally. In fact, because of complexity and cost, EDI is used by large businesses with custom industry metadata dictionaries. In contrast, a PIP is used by all businesses through standard industry metadata dictionaries.

Some of the PIP categories in RosettaNet are listed with [square brackets] indicating the number of PIPs defined in that category are listed in Table 3.

Table 3: List of RosettaNet PIPs

For example, the characteristics of PIP 3A4, manage purchase order (PO), are shown in Table 4. This PIP covers specifications for PO request, PO acceptance PO change and PO cancellation. These processes are used by ABC and DEF as customers of XYZ – their supplier – if these companies all operate in the IT or EC industries. Using PIP 3A4 specifications, a company is identified by its GTIN (global trade identified number) or by its defined DUNS number. (A DUNS number is an identifier that Dunn and Bradstreet uses to identify individual companies for credit worthiness enquiries.)

Important performance criteria are defined for each RosettaNet PIP. For example, PIP 3A4: Manage Purchase Order in Table 4 defines that: a supplier must acknowledge receipt of a 3A4 message within two hours; it must respond and agree or decline to act on the request within 24 hours – for which authorization is required. Secure transport, digital signature and secure sockets layer (SSL) security are all required.

Note that the specification indicates that non-repudiation is required. This is an important principle of RosettaNet. The definition of the RosettaNet specifications also established the legal nature of electronic documents, so that they have the same force in law as do paper documents.

Table 4: Content of RosettaNet PIP 3A4 Manage Purchase Order

For example, when a customer sends a paper purchase order to a supplier requesting specific products or services for fulfillment, and that PO is accepted by the supplier, a legal agreement is implicitly entered into by both parties. By mutual acceptance, the supplier has agreed to supply the requested products or services at the agreed price by the agreed date; by request, the customer also implicitly agrees to pay that price when the requested products or services are provided by the Supplier on that date.

The same legal principles also apply to RosettaNet electronic documents. For example, when a customer issues an electronic XML PO request for specific products or services to a supplier and the supplier responds with an electronic PO acceptance message, a legal agreement has been entered into between both parties. The PO request and PO acceptance are now legally binding on both parties. This is shown in Table 4 by specifying Non-Repudiation Required: YES.

RosettaNet, with its PIPs, message envelope formats and business document formats, is a full solution for e-business in the information technology and electronic components industries. Other more general solutions are available through software products such as Microsoft BizTalk Server and also ebXML software products from many vendors. These address message delivery with envelope formats as discussed next.

Electronic business XML is called ebXML. It defines the evolution of electronic data interchange applications and messages to XML. It is a joint effort by UN/CEFACT and OASIS (the Organization for the Advancement of Structured Information Standards).

ebXML is a worldwide standardization of XML business specifications to migrate EDI to XML. It defines a technical framework that enables XML to be utilized in a consistent manner for the exchange of all electronic business data.

The ebXML specification phase involved more than 3,000 organizations worldwide, meeting quarterly. It ran over an 18-month period from November 1999 through May 2001. The ebXML Specifications were first released in May 2001; they document ebXML for use by EDI and middleware software vendors. We saw that BizTalk uses the BizTalk Orchestration Designer to define process logic and workflow. The ebXML specifications originally assumed that business process logic would instead be defined in UML diagrams. These UML diagrams were to be used automatically to generate executable XML-based code in Business Process Specification Schema (BPSS). We will discuss BPSS in a later article on SOA. (We will see in the later SOA article that BPSS 2.0 specifications now refer to the use of business process modeling notation (BPMN), rather than UML, to define business process logic. BPMN will be covered in the SOA paper.)

ebXML was designed in part for use by large enterprises that use EDI. It was designed for use also by smaller companies, as discussed shortly. Large trading partners will likely continue to use their EDI software middleware vendors. But the advantage ebXML offers them is the low-cost capability of the Internet for message transfer, rather than the high-cost leased lines previously required for EDI. Conversion between EDI and ebXML is automatically provided by EDI software middleware vendors.

The ebXML message structure is very similar to that used for BizTalk messages. ebXML specifies an ebXML header envelope and an ebXML payload envelope as illustrated in Figure 12.

The header envelope contains the ebXML header document, with a manifest and details of the header, similar in concept to BizTalk. The payload envelope contains the actual XML document(s), called the payload documents by ebXML. BizTalk uses synonymous terms: business document and body of the message.

Figure 12: ebXML message envelope format. (© 2001 ebXML.org. Reprinted with permission.)

As discussed earlier, ebXML was designed also for use by small and medium enterprises (SMEs). Large EDI trading partners already do business with each other, but SMEs may not know of other SMEs that they want to trade with. Figure 13 shows how they can find each other.

Figure 13 shows typical use of ebXML by Company A and Company B. Company A decides that it wants to publish the availability of its products and services for online purchase using ebXML. For discussion purposes, we will assume that Company A is a bank (Bank A), which wants to offer its credit card authorization and processing services via ebXML. We will also assume that Company B wants to use ebXML for credit card authorization and processing from a bank. We will use the earlier example of XYZ, which now wants to implement an online store for its customers to visit. The XYZ online store decides that it wants to use ebXML for credit card authorization and payment. The following steps are referenced to Figure 13 using [brackets].

Bank A (Company A) requests ebXML specification details from ebXML.org: the ebXML Registry and Repository [1]. These specifications are sent by ebXML.org and are used by Bank A [2] to develop its credit card authorization and processing service details, for publishing to ebXML.org [3].

The XYZ Online Store (Company B) may be located anywhere in the world. It decides that it wants to find if any bank offers a credit card authorization and processing service using ebXML. It queries the ebXML.org Registry. It receives details of all banks that have registered to provide those services [4].

The XYZ online store selects Bank A and contacts it to exchange collaboration protocol agreements [5], based on the bank’s ebXML definition. The XYZ Online Store prepares its credit card authorization and processing service ebXML transaction requests based on the format and specifications published by the bank. Through specific development based on the Bank A specifications, the XYZ Online Store tightly binds itself to the bank. This was initially supported by ebXML with compile-time-binding: more recently ebXML has also specified support for run-time binding as with Web services. (The initial specification of ebXML was tightly-coupled based on compile-time-binding, with code based on the specifications of the ebXML service provider. In contrast, ebXML now – and Web services – are based on loosely-coupled run-time-binding with dynamic binding carried out in real time, as discussed in a later article.) Both companies commence business transactions together [6] – having established a valid business relationship based on the ebXML specifications.

Figure 13: Steps in using the ebXML repository. (© 2001 ebXML.org. Reprinted with permission.)

The steps in Figure 13 are as documented in the ebXML specifications. By using an ebXML middleware product, the bank supplies relevant details and BPSS code in XML for publication of the credit card authorization and processing service on ebXML.org (see steps [1]-[2]). The middleware package then publishes those specifications [3]. Similarly, also by using an ebXML middleware product, the online store asks the middleware product to “find” relevant suppliers of credit card authorization and processing services. Based on the response for all banks [4], and using the package to download the relevant BPSS XML code of interest, the XYZ online store contacts Bank A to enter into a collaboration protocol agreement – CPA [5].

As discussed previously, the ebXML Business Process Specification Schema 1.0 XML code was originally intended to be automatically generated from UML diagrams to define the collaboration and coordination of business processes in achieving a physical business interchange between parties.

XML Integration Server Concepts
We will now discuss the concepts and use of integration servers (also called “interaction servers”). These concepts are used by trading communities and also message broker software products to direct XML messages to the appropriate software code modules or programs, for processing relevant XML messages. The numbered steps in Figure 14 are keyed to the following discussion for XYZ Corporation.

Figure 14 illustrates processing involved in XML messages received across the firewall of an enterprise. On receipt, each XML message is examined by an integration server – such as used by message brokers or trading communities – to determine the content of the message and the application program or code module that is required to process it.

The XML document [1] is first located in a document repository. This indicates the relevant workflow steps [2] that are needed to process the received XML message. The XYZ document repository also specifies the data transformations [3] that are required to convert the XML message format to the input data format structure and data types expected by enterprise systems, for processing by the receiving application program or code module [4].

Figure 14: XML messages are examined for content and process workflow. They are transformed to the format expected by back-end enterprise systems. (eXcelon Corporation – now Sonic Software © 2001 Sonic Software. Reprinted with permission.)

Figure 15: XYZ workflow steps to accept an XML PO sent by ABC. (eXcelon Corporation – now Sonic Software © 2001 Sonic Software. Reprinted with permission.)

The concepts associated with integration servers are best described using an example based on XYZ and ABC, as discussed earlier. Figure 15 illustrates the relevant workflow steps to be followed by the XYZ order entry system on receipt of a purchase order sent by a buyer: ABC. The following steps are keyed to Figure 15. ABC in this figure uses the OAG (Open Application Group) PO XML message format.

We can see in Figure 15 that on receipt of the XML message [A], if the total value of the PO is <$10,000, the business rule used by XYZ is to immediately accept the order [B]. But if the total value of the PO is $10,000 or greater [C], the business rule requires the credit status of the buyer first to be checked [D] to determine whether the PO should be accepted, or rejected [E]. The buyer is then notified of acceptance or rejection by an XML message sent in response to the PO [F].

For accepted purchase orders to be processed as sales orders, Figure 16 shows that the quantity of each ordered item in inventory is checked by XYZ [G]. If insufficient quantity is available to fulfill the ordered quantity, XYZ may need to order sufficient quantities of each out-of-stock item from its own suppliers to replenish its inventory [H].

 Figure 16: XYZ workflow steps to process an XML PO that was sent by ABC. (eXcelon Corporation – now Sonic Software © 2001 Sonic Software. Reprinted with permission.)

When these items are received into inventory [I], or if sufficient quantity is already available, the buyer’s PO (from ABC) is processed [J]. An ASN is sent back to the buyer [K], together with a customer invoice [L] for the items that are supplied in satisfaction of the ABC purchase order.

Let us now look at the workflow processing for the buyer: ABC in Figure 17, with numbered steps now, keyed to the following discussion. Following transmission of the PO to XYZ earlier, the workflow engine at ABC examines each later received XML message and then carries out the steps detailed after the figure.

  1. The first message received by ABC is the delivery advice or advance shipping notice transmitted by XYZ in confirmation of processing the PO earlier sent by ABC to XYZ [1]. This ASN message is directed to the purchase order acceptance system for ABC processing [2]. The XML message indicates the date when the ordered items in the PO will be delivered into ABC inventory, and the agreed price between ABC and XYZ for each item. The copy of the PO that was saved originally by ABC is therefore updated accordingly with the ASN details [3].
  2. Later, the customer invoice transmitted by XYZ in XML is received by ABC as a supplier invoice [4]. This XML message is directed to the ABC accounts payable system for processing [5], which examines the supplier invoice to determine that: a) the referenced PO is valid; b) the ordered items have been delivered by XYZ; and c) they have been received into inventory by ABC. The items are checked to confirm that: d) the correct items were delivered in the correct quantity; e) they are in good order; and f) they are priced correctly [6].
  3. If the supplier invoice is correct as discussed in step 2, the ABC accounts payable system authorizes payment of the total amount of the supplier invoice to XYZ [7]. This is typically achieved by sending a wire transfer request message in XML to ABC’s bank, authorizing the automatic transfer of the invoiced amount due from the ABC bank account directly to the nominated XYZ bank account.

Figure 17: ABC workflow steps for receipt of the ASN and supplier invoice from XYZ. (eXcelon Corporation – now Sonic Software © 2001 Sonic Software. Reprinted with permission.)

The workflow process logic for each XML message is documented diagrammatically in a workflow diagram. In the past, this workflow diagram would then be printed so that a programmer could manually code the relevant process logic using various programming languages. (We used to joke that it was the programmer’s job to code in the “bugs”!)

Distinct from this past manual coding approach, most of today’s EAI workflow tools first provide facilities to test that the diagrammed logic in the workflow diagram is correct. This is similar to what we discussed with the Microsoft BizTalk Orchestration Designer. In the same way, after correcting identified logic errors in the diagrammed chart – instead of printing the diagram for manual coding – today’s EAI workflow tools automatically generate executable logic directly from the diagram, without requiring any manual programming.

The result is fast implementation with no coding errors or “bugs.” Furthermore, if any changes are later required to be made to the defined workflow, these changes can now be made at the diagrammatic level. The changed workflow diagram is tested again, and once more it is automatically regenerated as executable code. We will discuss these concepts further, when we cover service-oriented architecture in a later article.

Redundant Data Update Using EAI
We will now revisit the problem that was introduced earlier, with redundant data versions in the enterprise. We saw that data such as a customer address may exist in other versions also, such as client address and debtor address – as used by the sales department, the credit department and the accounting department. If a customer is also a supplier, we discussed that the supplier address and creditor address would also be redundant.

Figure 18 considers that each redundant address version must be kept up-to-date if any one version of this data changes. For example, if the customer address is changed in the order entry system, an address change notification form is printed out and sent by internal company mail (or by fax) to each part of the organization where the address exists redundantly. The relevant change address data maintenance transaction is then used to manually reenter the new address, thus updating the previous version of the address to the new, changed address.

The manual data maintenance activity in Figure 18, needed to keep each redundant data version up-to-date, involves redundant data entry of the changed address, updating the client address in the Credit Control System and the debtor address in the invoicing system for accounts receivable in the finance department. This is redundant work, with redundant staffing to do this redundant data entry. Also, the address change transaction used by each redundant data maintenance system may not use the same operating procedure or screen design for data entry; this will also require redundant training of staff to do this redundant data maintenance processing.

Figure 18: Address change notification form with manual reentry. (Adapted from Gartner © 2001 Gartner. Reprinted with permission.)

The redundant work, redundant staffing, redundant processing and redundant training are very expensive, time-consuming and error-prone. This is the high-cost consequence of redundant data. EAI now addresses this problem very efficiently and inexpensively, as discussed next in relation to Figure 19.

Figure 19: Address change notification using EAI avoids the need for manual reentry. (Adapted from Gartner © 2001 Gartner. Reprinted with permission.)

With EAI, rather than print then mail or fax the address change notification form to parts of the enterprise, in Figure 19 it is output as an XML address change notification message. This XML message is transmitted electronically to data maintenance address change systems throughout the organization using EAI middleware products such as BizTalk, or ebXML middleware products. The XML address change notification message can be delivered electronically and transformed to the input data format required by each address change system for direct input, without requiring any manual reentry.

The redundant work, redundant staffing, redundant processing and redundant training previously required for manual reentry is no longer needed. The electronic XML address change notification message is transmitted automatically in near-real time. The result is inexpensive data maintenance synchronization, with no errors introduced – a problem experienced with the previous manual data entry approach, as we discussed.

Using EAI methods and middleware products in this way, the redundant data version problem is addressed very efficiently and inexpensively. It converts the many redundant versions of the same data (which is bad), now to “replicated” data (which is good). This replicated data is now maintained and synchronized electronically in near-real-time, so that it is always kept up-to-date.

We will also revisit this same problem of redundant data again in a later article, when we discuss Web services.

We have covered many of the concepts and technologies behind enterprise application integration. Next month, I will discuss the EAI support provided by webMethods. This vendor provides the EAI Infrastructure used by other EAI vendors: by trading networks, by B2B vendors, by message brokers, message delivery vendors and other EAI middleware vendors. Many of these vendors license EAI technology from webMethods. They license EAI, B2B, reliable message delivery and envelope translation functionality needed for EAI. In fact, webMethods is the major EAI infrastructure vendor.

Summary of Enterprise Application Integration
I will complete this article with a brief summary of EAI concepts that I have covered:

  • Cost Reduction: The typical cost of manual entry and processing of a Purchase Order is $100 to $150, independent of the value of the items purchased. This cost typically drops to $8 to $10 using EAI for transaction processing. This is a powerful business driver for enterprise application integration (EAI).
  • Buyers and Suppliers: Trading communities (trading networks) help customers and suppliers to find each other, so they can do business together. This is most cost-effective when both organizations can eliminate the manual reentry of documents, and instead connect their back-end systems to each other automatically through EAI.
  • Document Delivery: Trading communities connect customers, suppliers and business partners to each other across the Internet – with reliable, guaranteed electronic document delivery. A variety of envelope formats and message protocols are used, with automatic transformation between them.
  • Message Formats: The major message formats include RosettaNet (for the IT and electronic components industries), with envelope formats based on BizTalk (from Microsoft) or ebXML (for use by large and small enterprises, via EDI and XML).
  • Industry Markup Vocabularies: These have been defined for most industries. These include: XBRL (for audit and financial reporting), IFX, OFX and FpML (for banking and finance) and HL7 (for healthcare). Other industry markup vocabularies are listed in the XML Catalog at OASIS.
  • Internet Yellow Pages: ebXML.org and UDDI.org both offer an “Internet Yellow Pages” capability, so that organizations can find each other easily – to do business together. This will be covered in a later Web services article.
  • XML Integration Servers: We looked at some concepts involved in XML message interchange, as carried out by integration servers. These concepts are based on workflow processes and also data transformation between different XML document formats.
  • EAI for Redundant Data: We saw how EAI can be used to maintain redundant data versions up-to-date, using XML documents to synchronize data changes throughout the enterprise in near-real time.

Next month, I will discuss EAI vendors and products.

  • Clive Finkelstein

    Clive is acknowledged worldwide as the "father" of information engineering, and is Managing Director of Information Engineering Services Pty Ltd in  Australia. He has more than 45 years of experience in the computer industry. Author of many books and papers, his latest book,  Enterprise Architecture for Integration: Rapid Delivery Methods and Technologies,  brings together the methods and technologies for rapid delivery of enterprise architecture in 3-month increments. Read the book review at http://www.ies.aust.com/ten/ten32.htm. Project references, project steps and descriptions are available from   http://www.ies.aust.com. Click on the  Projects link from any page. Clive may be contacted at cfink@ies.aust.com.

Recent articles by Clive Finkelstein



Want to post a comment? Login or become a member today!

Be the first to comment!