We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Bayes and Business Intelligence, Part 3 An Interview with Brad Efron of Stanford

Originally published March 17, 2009

As I started planning my BeyeNETWORK series on Bayes and business intelligence back in November 2008, I was beneficiary of some good old-fashioned luck. Friend Gib Bassett, a professor at the University of Illinois, Chicago, emailed an invitation to a series of upcoming lectures by Stanford statistician Brad Efron.

Now just about everyone in the statistical world knows of Efron as the author of the bootstrap, a methodology for using computation and sampling techniques for statistical estimation. The bootstrap, which changed the discipline of statistics 30 years ago, would have been exciting enough, but Efron’s presentations went beyond – to using emerging Bayesian techniques for solving knotty prediction problems. The lecture I attended, "Learning from the Experience of Others," was at once fascinating and foundational to my Bayesian education. After the talk, I introduced myself to Efron, who directed me to his web page for pertinent papers and presentations. It’s no accident that my Bayes and BI series referenced this work so fundamentally.

Brad Efron has enjoyed a long and productive career on the statistics faculty of Stanford University, the top-ranked such department in the country. His CV is a full eleven pages and notes his many Stanford faculty positions including professor, statistics department chairman and associate dean, as well as 141 publications. Brad’s given more special lectures, holds more association memberships, and has been awarded more prestigious academic honors, including the National Medal of Science, than I can count.

Efron has a variety of current statistical interests, including astrophysics, biostatistics, simultaneous inference and empirical Bayes. I particularly like many of his post 2000 papers and presentations, finding them more mathematically accessible, with a history of science wisdom. He’s also a very good writer – much better than he selflessly acknowledges.

Yet it’s Efron’s work on the bootstrap starting more than 30 years ago that established his standing at the top of the statistical profession. The basic concepts behind the bootstrap are remarkably simple. Statisticians are interested in making inferences about populations, but are almost always limited to small samples drawn from those populations for their analyses. The traditional connection between sample and population is a model, which often requires stringent assumptions about the behavior of sample observations and can involve quite complicated mathematics. The bootstrap, alternatively, treats the sample like a population, using computer simulation and random sampling techniques to repeatedly compute statistics of interest from the sample. With hundreds and even thousands of these replications, the distribution of the computed sample statistics can be quite well behaved and provide precise estimates of population parameters – for a fraction of the pain. Rather than wrestling with restrictive assumptions and arduous mathematics, the statistician can simply finesse calculations using computer power. Indeed, the bootstrap has helped change the practice of applied statistics and is now a staple in every statistician’s tool chest. There’s a consensus among academic statisticians I know that if there were a Nobel award for statistics, Brad Efron would have won for his seminal work on the bootstrap.

A second major interest for Efron is Bayesian analysis. In Bayes and Business Intelligence, Part 1, 1 discussed the frequentist versus Bayesian controversy in modern statistics. The approaches differ fundamentally in their characterizations of probability. Frequentists see probability as the objectively measured, relative frequency of an outcome over a large number of trials, while Bayesians view probability as a more subjective concept tied to an individual’s judgment of the likelihood of an event. Mainstream frequentists historically have struggled with the subjective prior probabilities of Bayesians, arguing they cannot be measured reliably. Bayesians, on the other hand, feel their approach is more pertinent and natural for real world problems. Empirical Bayes, the approach championed by Brad Efron, is a compromise that uses the abundance of data available today to move from subjective to objective priors, at the same time exploiting computational power to circumvent intractable mathematics. Today, statisticians can “enjoy the Bayesian omelet without breaking the Bayesian eggs.”

After posting Bayes and Business Intelligence, Part 1 , I sent an email to Brad with the article URL, hoping I hadn’t written anything too outrageous. He quickly opined that I hadn’t. We continued to correspond, and he graciously agreed to what I’m sure he thought would be another undistinguished interview. What follows is text transcribed from two after-class phone conversations we had in February/March. Rather than specific question and answers, I’ve distilled Brad’s wisdom into three broad areas. I hope readers enjoy Brad’s responses as much as I enjoyed our conversations.

Efron on the State of Statistics as an Information Science…

Statistics, the first and most successful information science, is a small discipline concerned with careful analysis of inference. Given that charter, statistics has important contributions to make to the sciences and has indeed gained in recognition over the last 45 years. The frequentist statistical theory that owes its origins to Fisher and Neyman more than 50 years ago gave us the information bound limits – What’s the best you can do? – that continue to serve us well today. If we abandon statistical inference, we’re doomed to reinvent it.

Statistics has enjoyed modest, positively sloped growth since 1900. There is now much more statistical work being done in the scientific disciplines, what with biometrics/biostatistics, econometrics, psychometrics, etc. – and business as well. Statistics is now even entrenched in hard sciences like physics. There are also the computer science/artificial intelligence contributions of machine learning and other data mining techniques. If data analysis were political, biometrics/econometrics/psychometrics would be “right wing” conservatives, traditional statistics would be “centrist,” and machine learning would be “left-leaning.” The conservative-liberal scale reflects how orthodox the disciplines are with respect to inference, ranging from very to not at all.

Data mining and machine learning are much more ambitious than traditional statistics but in many ways more superficial. Data miners are often too credulous for us trained statisticians. Perhaps that’s the case with the Wired article, "The End of Theory: The Data Deluge Makes the Scientific Method Obsolete." The neural net algorithms from psychology that were first promoted as a modeling panacea are now acknowledged as more limited.

While statistics is not front line in the machine learning world, it does have something to say. The field of statistical learning championed by Stanford colleagues Trevor Hastie, Rob Tibshirani and Jerome Friedman, halfway between statistics and AI, has provided a wonderful communication platform, bridging the gap between machine learning and statistics. The second edition of their outstandingElements of Statistical Learning book has just been released.

Friend and colleague, the late Berkeley statistician Leo Breiman, was somewhat of a handwringer about the state of our craft. Leo made many important contributions to the field, including popular machine learning algorithms Classification and Regression Trees (CART) and Random Forests. His provocative 2001 article "Statistical Modeling: The Two Cultures," contrasts the “models” of traditional statistics unfavorably with the black box algorithms developed by other disciplines, warning that the field of statistics is at risk of becoming irrelevant if it cannot accommodate to current prediction challenges.

A little handwringing can go a long way, but I don’t share Leo’s angst. The competition between machine learning and statistics can progress predictive science. And I sometimes think that the AI crowd is not critical enough – is a little too facile. The whole point of science is to open up black boxes, understand their insides and build better boxes for the purposes of mankind. Incidentally, the work of Leo Breiman will be recognized in an upcoming edition of the Annals of Applied Statistics.

Efron on the Impact of the Bootstrap…

(Laughing) First the bootstrap was wrong, and then the reaction was “I knew it all along!” Seriously, though at first criticized by Bayesians, the bootstrap was accepted remarkably well. There was much early theoretical interest, spawning over 10 years of research papers. Now the emphasis is on its applied usage in the disciplines. At this time, the bootstrap is a standard tool in the statistician’s arsenal, opening up predictions, standard errors, confidence intervals, etc. to ready computation – allowing practitioners to bypass the often-arduous mathematics. It provides for immediate statistical gratification and for quick communication. The computer does the theory; the statistician needn’t worry about the math.

Computation is now the third leg of a statistical triangle that also includes applications and mathematics. Actually, computation always has been third pole, but was static and impotent for so long. Electronic computation is now orders of magnitude faster. Though computation plays a big role, mathematical statistics hasn’t disappeared; it simply awaits the next golden age with a new Fisher or Neyman to reorder algorithms. A statistician from the 1950s would recognize the mathematics today – there’s really been little new theory in 50 years. What he wouldn’t recognize are the applications and the computational environments. It sure would be nice for a new wave of theoreticians to derive the information bound for models and predictions!

I don’t think of the bootstrap as an inflection point in statistical history. The work was really an extension of the jackknife methodology pioneered at Stanford in the ‘70s. At the same time, the bootstrap has been my single biggest hit. I’d love to find another. Perhaps the combination of empirical Bayes, simultaneous inference, and the bootstrap…

Efron on Bayesian Statistics…

(Laughing) I’ve always been a Bayesian; it’s Bayes practitioners that scared me. The Bayes movement is much more grounded now than it was 25 years ago. There seems to be more of a connection with real world problems and data, an evolution from subjective to objective. The tension between frequentists and Bayesians is not what it was years ago when Bayesians seemed too credulous. The competition between Bayesians and frequentists in real world applications should advance both, to the benefit of science and business.

Classical prediction methods such as Fisher's linear discriminant function were designed for small-scale problems, where the number of predictors is much smaller than the number of observations.  Modern scientific devices often reverse this scenario. A microarray analysis, for example, might include N = 100 subjects measured on p = 10000 genes, each of which is a potential predictor. An empirical Bayes approach where the prediction rule is estimated using data from all attributes might be optimal.

It’s pretty clear to me that statistics will play an increasing role helping 21st century sciences like biology, economics and medicine handle their large and messy data problems, and that an integration of frequentist and Bayesian approaches is probably ideal for meeting those data challenges.

One of the main responses of Bayesians to the objectivity demanded by frequentists is “uninformative priors” that are devoid of specific opinions. And the bootstrap, initially developed from a purely frequentist perspective, might provide a simple way of facilitating a genuinely objective Bayesian analysis. Though still suggestive, the bootstrap/priors connection merits further investigation.

Empirical Bayes combines the two statistical philosophies, estimating the “priors” frequentistically to carry out subsequent Bayesian calculations. Bayes models may prove ideal for handling the massively parallel data sets used for simultaneous inference with microarrays, for example. The oft-repeated structure of microarray data is just what is needed to make empirical Bayes a suitable approach.

I’m now prepping for a talk on "The Future of Indirect Evidence" that builds on the "Learning from the Experience of Others" presentation you attended. How is it, for example, that baseball player batting averages over the first month of the season can be used as reliable predictors of final season averages, and that Reed Johnson’s month one stats can help estimate Alex Rodriquez’s final season average? The answer lies in a statistical theorem that says the James-Stein empirical Bayes estimator beats the observed averages in terms of total expected squared error. Perhaps player agents can find use in that.


  • Steve MillerSteve Miller
    Steve is President of OpenBI, LLC, a Chicago-based services firm focused on delivering business intelligence solutions with open source software. A statistician/quantitative analyst by education, Steve has 30 years analytics and intelligence experience. Along the way, he has conducted health care program evaluations, performed and managed database services at Oracle, and helped build a public BI services firm as an executive with Braun Consulting. Steve blogs frequently on Stats Man's Corner at miller.openbi.com. He can be reached at steve.miller@openbi.com.

Recent articles by Steve Miller

 

Comments

Want to post a comment? Login or become a member today!

Posted September 28, 2010 by fjsamaniego@ucdavis.edu

A relevant reference: "A Comparison of the Bayesian and Frequentist Approaches to Estimation" by F. J. Samaniego (2010) New York: Springer.  This monograph proposes a framework in which one can judge when a Bayesian procedure stands to outperform its frequentists counterparts. An explicit solution to the "threshold problem", which identifies the boundary between good and bad Bayes procedures, is obtained in a variety of statistical settings. The monograph can be thought of as a treatise on "comparative inference".

--- Frank Samaniego

Is this comment inappropriate? Click here to flag this comment.