Business Intelligence Network - The Vision for BI and Beyond

This article proposes that there is a link between the philosophical studies of the nature of being and customer relationship management (CRM) studies of customer response. This link is seen most clearly when customer response is viewed through the metaphor of business as organism, rather than as machine. This approach is used to guide the design of powerful customer behavior models based on special abstractions of historical time series data. These temporal abstractions are keyed to the response date for each customer (rather than the calendar date) to reflect what customers did during the several months prior to the response to be modeled. This design is used to create a voluntary attrition model of households of automobile insurance policies. Results of this model show that only about 60% of the lift of the model over random is attributable to non-historical (static) variables. The other 40% of the lift is provided by the temporal abstraction variables.

Knowing How Our Customers Behaved Before They Acted

To be competitive in today's markets, we must capture and leverage information from historical detail records describing what our customer did in the past. This information can be very useful in defining patterns in the behavior of customers leading up to the decision to leave the company. For a given customer, the decision to leave the company did not happen in a vacuum. Many factors contributed to this decision, such as dissatisfaction with service, perception of the greater value of competitive goods and services, and changes in business needs. Some of these factors, such as customer satisfaction, can be tracked through customer care programs. However, most factors that contribute directly to attrition cannot be captured and stored in corporate databases. The only way to reflect these attrition variables is to relate them to customer behavior patterns that can be tracked from data in the data warehouse. The pattern of historical information of customers who have left the company can be used to predict which present customers have a high probability of leaving in the near future. How is this possible?

Transforming Corporations into Business Ecosystems – The Path to Customer Fulfillment

Ever since the Industrial Revolution, western society has tended to view the world as a machine, composed of components that functioned like cogs, wheels and springs.

Newton formalized this approach in science. However, it only worked within the range of Newton’s instruments. Later discoveries by Einstein (relativity) and quantum physicists caused the Newtonian concept of the world to fall to pieces!

Business also picked up on this metaphor in the Industrial Revolution. The automobile assembly line of Henry Ford was viewed as the paragon of efficiency.

As long as the product was relatively simple in organization, this metaphor appeared to work. An efficient business became defined in terms of:

• a “well-oiled machine”
  
  
• “having momentum”
  
  
• “gaining steam”
  
  
• “firing on all eight cylinders”

The primary business unit became the corporation. The prevailing attitude was “us against them.” Only the strong competitors survived. For these corporations, the primary business activity was production. It was expected that revenue would be maximized as production was optimized. Generations of operations research practitioners sought to optimize processes that would maximize business revenue.

With the advent of fast computers, flexible communications and (currently) the Internet, a new business paradigm has emerged – the business ecosystem (Inmon, et al, 1998). Moore (1999) maintains that real competition in these business ecosystems is not dead (actually, it is intensifying), it has just changed its expression. The old expression of competition pitted offers and markets against each other. The products improved as companies listened to customers and made the products fit their desires. The problem with this approach is that it ignores the environment and the system in which those offers and markets are embedded. It also ignores the great benefit that can come with coevolution with other “competitors” to satisfy customers more than if they operated separately. Moore stresses the importance of the environment and the system in which our businesses are enmeshed. This emphasis points also to the need to consider the system’s effects in our analyses of customer behavior.

As businesses became more complex, the machine metaphor began to break down. In both science and busi-ness, it became increasingly obvious by the 1980s that we had to begin to look at the world in a different way. In these increasingly complex systems, there seemed to be important properties that did not emerge until the system was complete and operating as a whole. These emergent properties often controlled the major responses of the system. These influences are causing a profound shift in science and business toward viewing the world as organism!

Petzinger (1999) remarks that the key characteristic of modern civilization is that of economizing, and that our genes are programmed for business. This view of business as “organism” flowed out of the bionomic principle (Rothschild, 1990):

There is a parallel between the response of natural systems to rapid environmental change and the re-sponse of business systems to rapid technological change.

From this principle, it is argued that our view of the world-as-machine greatly hinders us from economizing very well in this age of rapid technological change. Why? Because the rules keep changing faster than our machine-like business systems can accommodate. Perhaps it is time for a “new” science to help us understand life in the midst of rapid change (Rothschild, 1990).

Many companies have been influenced by bionomics to change the basic way they do business. Examples of these companies include Microsoft Corporation, Herman Miller, Fingerhut, Dell Computer and Yahoo!

Microsoft. The Microsoft Corporation helped to shift the primary business activity from production to customer fulfillment. To do this, they built business webs that functioned like food webs in natural eco-systems. These business webs combine producer, shipper and retail outlet functions of many companies to create a supply chain. Efficient management of the supply chain can increase efficiency and profits of all members of the business web.

Herman Miller (an office furniture maker) has established a business web on the Internet to share information with their suppliers and to remain in close touch with their customers. This coordination between suppliers, manufacturer and suppliers turns the supply chain upside down. Formerly, the chain was driven by supply; business webs turn the supply chain into a demand chain.

Dell Computer: Probably the king of demand chain marketing is Dell Computer. Dell does not create the computer from pieces until the customer orders it. Afterward, Dell provides 24-hour on-site support of their computer systems. This combination of demand chain processing and excellent customer support made Dell into a billion dollar company in only 3 years.

Fingerhut: Fingerhut once thought they were a catalog company. They invested more than $200 million in modern warehouses; but by 1997, more than half of the warehouse space was empty. Fingerhut created a new division, Business Services, to occupy some of this excess space. The new division contracted out Fingerhut’s back-office operations and began providing other companies with data mining, order processing and other services. Fingerhut’s Business Services expanded into the Internet. Today, Fingerhut is the largest end-to-end direct-to-consumer business web in the country, managing inventories and orders for over 20 web retailers (e.g., Pier 1 Imports, Children’s Place).

Free Chess: Free chess playing sites are provided by Microsoft and Yahoo! This provides enjoyment and recreation to thousands of players worldwide. What do Microsoft and Yahoo! get out of this? They install cookies and customer tracking products on players machines, which permit those companies to harvest a rich picture of what players are doing on other sites. This information can be used by Microsoft and Yahoo! to gain insights to customer behavior and to provide information that can be used to tailor ads for specific products to their players. The companies gain competitive advantage by fulfilling their customers’ desires (playing chess at any time).

CRM in Business Ecosystems

In the freewheeling business of today, companies try to build CRM programs that aim to create the same kinds of relationships with their customers. In order to build these relationships, companies must learn to understand their customers. To understand their customers sufficiently to build effective customer relationships, marketers must:

• Learn how to identify the right set of customers to do business with (segmentation).
  
  
• Learn how to identify valued customers.
  
  
• Learn how to recognize danger signals in their data relating to customer behavior that, if unchecked, might lead to decisions to leave the company.
  
  
• Use segments defined by attrition probability algorithms to strengthen and maintain relationships with valued members of the existing customer base.

The key principle in this approach is that the most powerful predictors of customer behavior in the future are customer behavior patterns in the past. Other customer characteristics are important also in defining patterns of customer behavior (i.e., demographic and firmographic information). However, unless we include in our models of customer behavior the patterns of past customer behavior related to their future actions, they will not be very powerful predictors of what customers actually do. When combined with the more static customer information gathered by businesses in their day-to-day operations (e.g., the date a business started business), companies can take a quantum leap forward in understanding the customer and improving customer loyalty.

Differences Between Static Measures and Evolutionary Measures

The key difference between historical behavior patterns and relatively static characteristics of customers is that historical patterns enable us to track the development of the decision to leave, rather than just the decision itself. These evolving behavior patterns are very organic in nature and are driven by a number of significant nonlinear events (NLEs). Farrel (1998) maintains that bursts of customer demand (or “anti-demand” like attrition) are driven by these NLEs. The evolutionary nature of these NLEs renders them much richer in predictive value than static characteristics alone, because they can capture the mood of the customer, preferences, attitudes and many clues that help you to understand why the customer did what he did. Some static characteristics are certainly related to the attrition decision, but they tell only a part of the story. In order to see the other half of the story, variables that express this development of the decision to leave the company must be added. This is a very organic view of customer behavior similar to the way biologists view the complementary effects of intrinsic (organism-based) and extrinsic (environmental) influences on organism response. This viewpoint represents a dramatic shift in mind-set from the traditional way that many companies view their data. Traditional approaches to modeling customer response relied principally on relating the response decision to specific data elements (variables) available in the database. This approach to the search for truth can be traced all the way back to Aristotle of ancient Greece.

Aristotle believed that the true being of things could be discerned by what the eye could see, the hand could touch, etc. He believed that the highest level of intellectual activity was the detailed study of the tangible world around us. Only in that way could we understand reality. Based on this approach to truth, Aristotle was led to believe that you could break down a complex system into pieces, describe the pieces in detail, put the pieces together and understand the whole. For Aristotle, the “whole” was equal to the sum of the parts. This nature of the “whole” was viewed by Aristotle as very machine-like.

Science gravitated toward Aristotle very early. The nature of the world around us was studied by looking very closely at the physical elements and biological units (species) that composed it. As our understanding of the natural world matured into the concept of the ecosystem, it was discovered that many characteristics of ecosystems could not be explained by traditional (Aristotelian) approaches. For example, in the science of forestry, we discovered that when you cut down a tropical rain forest on the periphery of its range, it may take a very long time to regenerate (if it does at all). We learned that the reason for this is that in areas of stress (e.g., peripheral areas), the primary characteristics necessary for the survival and growth of tropical trees are maintained by the forest itself! Nutrient cycling requires living trees as the source of nutrients because there is almost none in the soil (high rainfall leaches them out of reach of the tree roots). The forest canopy also maintains favorable conditions of light, moisture and temperature required by the trees. Removing the forest removes the very factors necessary for it to exist at all in that location. In order to understand the failure of Aristotelian philosophy for completely defining the world, we must return to ancient Greece and consider Aristotle’s rival, Plato.

Plato was Aristotle’s teacher for 20 years, and they both agreed to disagree on the nature of being. While Aristotle focused on describing tangible things in the world by detailed studies, Plato focused on the world of ideas that lay behind these tangibles. For Plato, the only thing that had lasting being was an idea. He believed that the most important things in human existence were beyond what the eye could see and the hand could touch. Plato believed that the influence of ideas transcended the world of tangible things that commanded so much of Aristotle’s interest. For Plato, the “whole” of the essence of being was greater than the sum of its tangible parts.

Plato agreed that there was indeed a reality to the world of our senses. However, he believed that there was a deeper reality of idea behind it. He called these central ideas forms. In his classic work, The Republic offered his analogy of the cave to explain the difference between these two worlds. Imagine, he said, that you are a slave chained in a cave and bound to look only in the direction of the cave wall. You can see shadows on the wall, moving, diverging and converging in 2 dimensions. These shadows, Plato argued, have a form of reality, but there is a higher form of reality that causes the images. If you were permitted to turn around and look at what causes the shadows, you would see a deeper reality in 3 dimensions.

The concept of the nature of being was developed initially in western thinking upon a Platonic foundation. Platonism ruled philosophy for over 2,000 years – up to the Enlightenment. Then, the tide of western thinking turned toward Aristotle. This division of thought on the nature of being is reflected in many of our attempts to define the nature of being in the world. We speak of the difference between “big picture people” and “detail people.” W contrast “top-down” approaches to organization versus “bottom-up” approaches. These dichotomies of perception are little more than a rehash of the ancient debate between Plato and Aristotle. Within the current emphasis on Aristotelianism in our culture, is it time to recover some of the benefits of pre-enlightenment thinking? For some in business today, the answer is yes! For modern Platonists (actually, Neo-Platonists), the significance of the deeper reality in the world points to a deeper cause for human action – human nature.

How Can Human Nature as Viewed Through Plato Help in Modeling Customer Response?

If human nature is a common basis for human action, then to predict the action of customer response, we must model human nature. We must focus on variables available to us in our databases that reflect some aspect of human nature that leads to the response. These variables might include:

• Historical customer care data
  
  
• Historical use of company services
  
  
• Historic billing revenue data
  
  
• Historic contract data
  
  
• Selected demographic data

How Can We Reorganize Our Data to Reflect Motives and Attitudes?

The key to successful customer response modeling is to associate with each customer a historical time series of fields (selected from those listed above) that in some way reflect motives and attitudes that caused the customer decision. These motives and attitudes flowing out of our human nature are the reality behind the “shadows” of the action. In order to see the deeper reality of what causes these shadows, we must turn around, so to speak (like those in Plato’s cave), and look at the data in a different way. We must abstract information from the time series of customer response in a form that is related to the customer action to be modeled. These abstrac-tions are called “temporal abstractions.”

The use of temporal abstractions has attracted widespread interest in medical and pharmaceutical informatics for predicting patient responses (Haimowitz and Kohane, 1996; Kahn, et al, 1991; Kattan, et al, 1997). Temporal abstractions are one type of data abstraction used to map data elements to some context environment. Data abstractions can be classified into 4 groups (Lavrac, et al, 2000):

• Qualitative abstraction – a numeric expression is mapped to a qualitative expression. For example, in an analysis of teenage customer demand, compared to that of others, customers with ages between 13 and 19 could be abstracted as a value of 1 to a variable “teenager,” while others are abstracted to a value of 0.
  
  
• Generalization abstraction – an instance of an occurrence is mapped to its class. For example, in an analysis of Asian preferences, compared to non-Asian, listings of “Chinese,” “Japanese” and “Korean” in the race variable could be abstracted to 1 in the Asian variable, while others are abstracted to a value of 0.
  
  
• Definitional abstraction – in which one data element from one conceptual category is mapped to its counterpart in another conceptual category. For example, when combining data sets from different sources for an analysis of customer demand among African-Americans, you might want to map “Caucasian” in a demographic data set and “White Anglo-Saxon Protestant” in a sociological data set to a separate variable of “Non-black.”
  
  
• Temporal abstraction – a variable in a time domain with one reference is mapped to a time domain of another reference. This is the form in which temporal abstractions are discussed below. The first three types of data abstractions are usually referred to by data miners as forms of recoding. The fourth type, temporal abstraction, is not commonly used. However, the methodologies of several data mining tool vendors do have forms of temporal abstractions integrated into their design.

What is a Temporal Abstraction?

Modeling customer behavior with temporal abstractions involves rearranging all the modeling variables to more clearly reflect patterns of change in the customer response variable. Then, the modeling tool can easily recognize the pattern that exists between the response variable and various states of predictor variables in the past with respect to the response variable. These time series representations of each predictor variable are a form of temporal abstraction.

This rearrangement is analogous to sliding the calendar time series of records for individual customers (following some sort of algorithm) until the pattern emerges. This operation is diagramed in Figure 1 using the analogy of an abacus.

Figure 1: Pattern Emergence Facilitated by a Temporal Abstraction

Figure 1 displays fields in six customer records from a telecommunications company lined up like beads on the abacus. The data on the left abacus represents information stored for monthly call duration extracted from multiple records in the database. This arrangement is similar to the format of the data extracted from databases into flat files to be submitted to the modeling tool for analysis. In the default configuration (left hand abacus), the yellow beads (a given state in the time series) are scattered all over the abacus. The diagram on the right of Figure 1 shows the rearranged data. Now, the yellow beads are lined up. The pattern emerges to the physical senses of our eyes, and likewise to the mathematical senses of the modeling tool.

The same approach can be used to model customer fraud or propensity-to-buy to serve cross-selling and up-selling campaigns. Thus, this approach to customer behavior modeling will permit us to see the “shadows” of customer behavior (following Aristotle), and reflections of the causes (the deeper reality) of this behavior (following Plato). Such a combined perspective on the nature of customer behavior can provide much more power-ful models than those based on one perspective alone.

Example

In order to test the relative contribution of temporal abstraction and static variables, voluntary attrition among customers of a large insurance company was modeled separately with each of the variable sets. The temporal abstractions were created by taking quarterly snapshots of policy records for a given household. The snapshots represent temporal objects in a temporal database (Jensen, et al, 1996). The temporal abstractions represent keys of this derived temporal database, in which the temporal tuples are the response quarter and a given quarter prior to the response. These snapshot temporal abstractions follow Snapshot Dependency Theory as extended by Wijsen et al (1993) and formalized by Wijsen (2001), and they represent keys for a sequence of snapshot relations in the household insurance policy history indexed in reference to the response quarter.

Methods

Parameteric statistical analysis operates on a statistical construct designed in the 1920s by Sir R.A. Fisher for analyzing and comparing medical data. Unfortunately, in order to make his methods work in a world of noisy nonlinearity and factor interactions, Fisher and his followers had to make a number of assumptions and add several compensating terms to their analysis (to account somewhat for nonlinearity and total factor interaction).

Among the assumptions of this “Parametric Model” were:

1. The data distribution is known, and most analyses assume a normal distribution.
   
   
2. The predictor variables exhibit a linear, additive effect on the total relationship to the predicted variable (Y).
   
   
3. All the variables are independent from each other in their effect on the predicted (dependent) variable. This means that there is not significant collinearity among the variables (two variables related to each other).
   
   
4. There is no significant heteroscedasticity (the variance is the same throughout the range of a variable).

But, what if the variables are highly nonlinear? What if they are multiplicative, rather than additive? They are in forest modeling analyses, for example. Generations of business school graduates have been taught that one could successfully apply parametric statistical analysis to business data, without warnings about the likelihood of violation of the assumptions of the Parametric Model, or any guidance of the effect of those violations on the credibility of the solutions. Both Mark Twain and Benjamin Disraeli quipped critically at the way statistics were used to “prove” issues.

“There are three kinds of lies: lies, damned lies, and statistics” – Mark Twain, Autobiography, Ch. 29;

“There are lies, damned lies, and church statistics” – Benjamin Disraeli, quoted in The Great Quotations by George Seldes

Fisher designed his statistical tools for use in the medical world to permit different researchers to analyze the same data and get the same results. Previous (Bayesian) statistical methods with their subjective "priors" did not lend themselves well to that end. To make these methods work, scientists had to do controlled experiments, holding all variables constant and varying the treatment of one variable at a time. Results were compared to a "control" group with no treatments. Laboratory conditions of temperature, light, moisture, etc. often had to be held constant because the physics of variable response might be affected by the environment. These highly controlled conditions are almost never found outside of a laboratory, but business analysts used these methods anyway.

Machine learning technology (particularly, neural nets) developed in the artificial intelligence (AI) community was not based on calculation of “parameters” like standard deviation. Modern neural nets do not depend on data drawn from a distribution of any particular kind (e.g., normal distribution). Patterns in data sets can be modeled directly in the form of weights assigned to each input variable. It is for this reason that machine learning technology is superior to parametric statistical analysis for modeling of highly nonlinear patterns in data. Both approaches are used in data mining analyses; but most often, the final modeling is done with a machine learning technique (e.g., a neural net)

An SPSS-Clementine Version 5.1 back-propagation neural net was selected for analyze the pattern of nonlinear events in the data set. Data preparation of the temporal abstraction variables was done with a C-program outside the data mining tool (at that time, no data mining tool could do that). A Clementine stream was designed to input data, train the neural net and score the hold-out data set with the trained model (Figure 2).

Figure 2: A Clementine Visual Programming Stream Used to Train a Neural Net and Score a Data File

A second Clementine stream was used to aggregate and decile the scored, and to create the lift curves (Figure 3).

Figure 3: Clementine Stream Used to Create the Lift Curves

Results

The cumulative lift diagram (Figure 4) is created by plotting the cumulative response(s) along with the cumulative response that you would expect from random selection. The random number expected for customer re-sponse in each decile is 10% of the total.

Figure 4: Cumulative Lift Curve for Disenrollment

Comparison of Static and Evolutionary Effects

A model was used to score the hold-out data set using static plus temporal abstraction variables; another model used only the static variables. Therefore, the cumulative total expected from random selection alone shown in Figure 5 rises from 10% (276.2 customers) in decile 1 to 100% (2,762 customers) in decile 10. Figure 5 shows that only about 60% of the lift (extension of the bar above random for a given decile) was due to the static variables. The rest of the lift is provided by the temporal abstraction variables.

Figure 5: Lift Curve for Static Plus Temporal Abstraction Variables

Conclusions

1. The biological metaphor appears superior to the machine metaphor in helping us understand the causes of customer behavior in business.
   
   Customers are biological entities that respond in a biological manner. This manner is the result of the complex interaction of many factors that operate as a system to influence the nature of the response. It seems reasonable to expect that the only way to create highly predictive models of customer behavior is to express some degree of this complex interaction in the design of the modeling methodology.
   
   
2. There is a link between the philosophical studies of the nature of being and the business studies of customer response.
   
   We can gain valuable insights into the nature of customer response by relating it to the classical dichotomy of thought expressed in the works of Plato and Aristotle. For over 2,300 years, the “big picture” approach to truth used by Plato was contrasted with the study of detail by Aristotle. They were viewed as opposing approaches to truth that grew into opposing schools of thought in virtually every natural science discipline. For example, the autecologists (affected dominantly by Aristotle) warred against the synecologists (affected dominantly by Plato) during the first part of the 20th century. The autecologists sought to understand biological systems by understanding their functions at the chemical, physical and mathematical level. Synecologists, on the other hand, defined biological systems more in sociological terms. The autecologists had a hard time “seeing the forest for the trees,” while the synecologists had a hard time “seeing the trees for the forest.” It became obvious to ecologists only during the last 30 years that we need both perspectives to generate a complete understanding of natural ecosystems. Likewise, we must include both “Platonic” and “Aristotelian” variables in our models of cus-tomer behavior in business systems.
   
   
3. Modeling with static variables alone will detect only part of the signal of customer response.
   
   Most attempts at modeling customer response are confined to analyzing historical variables and their trans-forms as if they are time-independent influences on customer response. Occasionally, time series analysis is applied to capture time dependencies. However, time series analyses (like all parametric methods) suffer from many assumptions that are not satisfied in business data (e.g., linear additivity, variable independence, membership in a specific distribution such as a normal distribution). Therefore, much analyses of historical data treats the data elements as if they were static variables with no time series attributes. For example, call durations for March, April and May are part of a time series of historical data for a given customer, but they are usually submitted to the modeling tool without regard to their sequence or relationship to the point in time of the customer response. This relationship constitutes a time dependency between the response date and the sequence in the state of each variable prior to that date.
   
   
4. Temporal abstractions provide a rich source of predictability for customer response.
   
   These temporal abstractions can provide insights into why the customer acted and represent the “other half” of the story of customer response that we can see in our databases. Temporal abstractions (and trend variables cal-culated from them) permit the data mining tool to capture much of the signal of customer response resident in the time series of the historical data.

References:

Farrel, W., 1998. How Hits Happen. Harper-Business.

Haimowitz, I.J., and Kohane, I.S. 1996. Managing temporal worlds for medical trend diagnosis, Artificial Intelligence in Medicine 8(3):299-321.

Inmon, W.H., Imhoff, C., and Sousa, R. 1998. Corporate Information Factory, Wiley Computer Publishing, pp 1-11.

Jensen, C.S., Snodgrass, R.T. and Soo, M.D. 1996. Extending existing dependency theory to temporal databases, IEEE Transactions on Knowledge and Data Engineering, 8(4):563-581.

Kahn, G.M., Fagan, L.M. and Sheiner, L.B. 1991. Combining physiologic models with symbolic methods to interpret time-varying patient data, Meth. Inform. Med. 30:167-178.

Kattan, M.W., Ishida, H., Scardino, P.T. and Beck, J.R. 1997. Applying a neural network to prostate cancer survival data, Intelligent Data Analysis in Medicine in Intelligent data analysis in medicine and pharmacology (Lavrac, N.,  Keravnou, E. and Zupan, B. editors),  Kluwer,  1997, pp 295-306.

Lavrac, N., Keravnou, E. and Zupan, B. 2000. “Intelligent Data Analysis in Medicine”, White Paper, Faculty of Computer and Information Sciences, University of Ljubljana, Slovenia.

Moore, J. F. 1996. The Death of Competition: Leadership & Strategy in the Age of Business Ecosystems, Harper-Business. pp 1-23.

Petzinger, Jr. T., The New Pioneers: The Men and Women Who Are Transforming the Workplace and Marketplace, Simon
 & Schuster, 1999. p 23.

Plato. 1979 ed., The Republic, Penguin Classics.

Rothschild, Michael. 1990. Bionomics – Economy As Ecosystem, Henry Holt.

Wijsen, J., Vanderbulcke, J., and Olivie, H. 1993. Functional dependencies generalized for temporal databases that include object-identity. Proceedings of the International Conference on the Entity-Relationship Approach, Arlington, TX. pp 100-114.

Wijsen, J. 2001. Trends in Databases: Reasoning and Mining. IEEE Transactions on Knowledge and Data Engineering, 13(3):426-38.