We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Bayes and Business Intelligence, Part 2 Learning from Bayes

Originally published February 3, 2009

Bayes Law Review

Bayes and Business Intelligence, Part 1 introduced the concepts of conditional probability and Bayes Law. In its simplest form, Bayes Law can be summarized as follows: If E is an event or hypothesis of concern and D is data or evidence, we are interested in P(E|D), the probability of  hypothesis E given or conditioned on evidence D.  Using basic laws of probability, P(E|D)  can be calculated as:

P(E)*P(D|E)/(P(D|E)*P(E) + P(D|~E)*P(~E)),

where ~E and ~D represent not event E and not data D, respectively.

P(E|D) is often called the posterior probability,
while P(E) is the prior probability,
P(D|E) is the likelihood function,
and the ugly right-side denominator P(D|E)*P(E) + P(D|~E)*P(~E ) is called a normalizing factor, though often lumped with P(D|E) as the likelihood function.

In other words, the:

posterior probability = prior probability*likelihood function/normalizing factor.

What makes this mumbo-jumbo pertinent is its potential for helping business intelligence (BI) realize its charter to facilitate organizational learning and decision making. A company can, for example, assess the posterior probability of an important business outcome given a shift in strategy or operations by establishing the known prior probabilities and wrestling through a likelihood function. The calculated posterior probabilities from step one then become the priors for step two, and the posteriors ~= priors*likelihood cycle repeats – promoting adaptive learning.

Bayes Law or Bust

The use of Bayesian analysis has increased significantly in science and business over the last 10 years. Bayesians are not shy and tend to be an enthusiastic lot, confident in their approach. Michael Lavine’s informative case study comparison of Bayesian and frequentist methods for estimating cancer incidence – What is Bayesian statistics and why everything else is wrong – is only partly tongue in cheek.

Esteemed Stanford statistician and Bayesian convert Brad Efron offers a more sobering view of the challenges facing Bayesians

“Now Bayes rule is a very attractive way of reasoning, and fun to use, but using Bayes rule doesn’t make one a Bayesian. Always using Bayes rule does, and that’s where the practical difficulties begin: the kind of expert opinion that gave us the prior odds one-third to two-thirds usually doesn’t exist, or may be controversial or even wrong. The likelihood ratio can cause troubles too. Typically the numerator is easy enough, being the probability of seeing the data at hand given our theory of interest; but the denominator refers to probabilities under other theories, which may not be clearly defined in our minds.”

The handling of prior probabilities has always been a lightning rod for Bayesians, especially given that: “The same data (likelihood) can lead to different conclusions depending upon other available information (priors).” Pre-season polls for college football are a good illustration of the potential for priors to contort down-the-road posterior rankings, much to the chagrin of many fans. Business Bayesians often are unapologetic of their subjective priors, arguing that the measuring stick for them is performance, not objectivity.  Those closer to mainstream statistics might opt for uninformative priors that serve to minimize the impact of P(E) on posteriors. An exciting approach championed by Efron is empirical Bayes, which uses the wealth of data often now available in business/science to independently estimate priors.

Bayesian likelihood presents different challenges for analysts, often posing intractable mathematical puzzles, solutions for which could not even be approximated until the emergence of desktop computing over the last 25 years. An excellent article by political scientist Simon Jackman provides understandable explanations for the latest methods to estimate likelihood functions, using interesting political polling and roll call illustrations. The best current approach appears to be the Monte Carlo, Markov Chain sampler (MCMC), which combines computer simulation with probability calculations.

Bayes Law Learning in Business/Science/Government

There is no shortage of pertinent Bayesian applications to provide guidance for the willing BI analyst. A sampling:

  • Machine Learning – algorithms such as the simple Naïve Bayes1 or the more complicated Bayesian Network1 and Bayesian neural networks  can be used for classification and prediction problems.

  • Expert Systems – first enabled by Bayesian Networks, the Decision Theory and Adaptive Learning group of Microsoft incorporated Bayesian learning into a number of products, most notably the “paperclip” Microsoft Office Assistant.

  • Risk Management – an inherently subjective Bayesian problem that manages the assessment, forecast and inference involved with lending, portfolio management and other insurance problems.

  • Fraud Detection – uses Bayesian learning and over-time patterns of behavior rather than individual events to predict fraud.

  • Diagnosing Network Problems – Nokia uses a commercial tool deploying Bayesian Networks as a prototype for diagnosis of its mobile infrastructure.

  • Spam – an open source anti-spam email filter, POPFile, makes use of a simple Bayesian component that “learns” how to recognize/differentiate spam from non-spam.

  • Medical Diagnosis – can be used to diagnose specific diseases based on ailments and other input factors, and is often a worthy replacement for rule-based diagnoses.

  • Clinical Trials – can drive adaptive randomization that promotes progressive allocation of patients to treatment doses found more effective, perhaps requiring fewer expensive patients and learning earlier if a study should be scuttled for lack of effect.

A current favorite involves Bayesian foreign intelligence. The analyses found that the use of Bayesian calculations for predicting hostilities in the Middle East were superior to expert opinion for non-events. The Bayesian calculations also arrived at their low probability of trouble predictions more quickly than experts.

Bayes Law and the Mind

Adding to the growing legitimacy of Bayes Law is the traction that the posteriors = priors*likelihood learning equation is getting as a plausible model of cognition in psychology. In a study that tested whether certain cognitive tasks were executed in accord with Bayes Law, psychologists from Brown University and MIT conducted laboratory experiments, giving participants nuggets of “prior” information, then asking them to draw conclusions – to make posterior judgments. A first example question went something like: A movie has grossed $X million to an unknown date (where X is either 1, 6, 10, 40, 100), how much was the total gross? A second question: How long would you estimate the life span for an 18 year old (choices 18, 39, 61, 83, and 96 years)?  Among the findings: Participant judgments for life spans, movie grosses and several other like questions were indistinguishable from optimal Bayesian predictions based on the empirical prior distributions. Though there wasn’t total alignment of findings with Bayesian predictions, the results are certainly suggestive and warrant additional scrutiny.

Further grist to the Bayesian mill is provided by research being conducted by neuroscientists proposing the “Bayesian brain” – a probability machine that repeatedly makes predictions about the world and then updates them based on what it senses. According to recent neuroscience theory, many brain processes look to minimize “free energy,” the difference between total and useless available energy. This minimization is a potential explanation for the constant updating of the brain’s probabilities. Tests for the free energy minimization principle based on computer simulation of neurons passing signals suggest neurons adjust to new information in a manner similar to that predicted by Bayes Law. Evidence from magnetic resonant imaging during visual tasks provides preliminary support for the theory.

Conclusion

It’s hard to look out at the statistical and business intelligence worlds today and not be exposed to Bayes Law in a significant way. Bayesian thinking directly addresses the critical question of how information from data modifies our beliefs about hypotheses and helps us make decisions – and is thus keenly pertinent for business intelligence. The cycle of Bayesian thinking – priors ’ posteriors ’ new priors – models how we learn, and is now supported by research in psychology and neuroscience. Moreover, Bayesian analysis is ubiquitous and prevalent in just about any area of learning and decision making. And advances in both the theory and computational efficiency of Bayesian methods are ongoing, making them ever more accessible. Prudent companies and BI organizations should adapt and adopt Bayesian analytics to be a core asset of their business intelligence portfolios.

End Note:

  1. Ian H. Witten and Eibe Frank. Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufman Publishers. 2005.

Referenced Work:

Andrew Gelman, John B. Carlin, Hal S. Stern and Donald B. Rubin. Bayesian Data Analysis. Chapman & Hall/CRC. 2004.

 

 

  • Steve MillerSteve Miller
    Steve is President of OpenBI, LLC, a Chicago-based services firm focused on delivering business intelligence solutions with open source software. A statistician/quantitative analyst by education, Steve has 30 years analytics and intelligence experience. Along the way, he has conducted health care program evaluations, performed and managed database services at Oracle, and helped build a public BI services firm as an executive with Braun Consulting. Steve blogs frequently on Stats Man's Corner at miller.openbi.com. He can be reached at steve.miller@openbi.com.

Recent articles by Steve Miller

 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!