BeyeNETWORK - The Global Vision for BI and Beyond

I had my first comprehensive test as an interview journalist with Super Crunchers author Ian Ayres, and botched it completely. Oh, I meticulously planned the questions and properly taped Ian's informative responses to our 75-minute cell phone discussion – Ian was traveling as usual, this time to Washington, D.C., for one of his many advocacy projects. After the call, I was listening to the first tape when my daughter phoned for me to pick her up from school. Of course, I inadvertently hit the record button rather than stop when I took her call. Smooth. I probably lost about 20 minutes of Ian's wisdom. The good news is that Ian was very understanding, agreeing to re-interviews as needed. As luck would have it, there were several recurring themes to Ian's responses that came out in multiple questions, so there'd have been some redundancy anyway. That's my story and I'm sticking to it.

With a Yale law degree and a Ph.D. in economics from MIT, Ian certainly has more than enough intellectual horsepower for his joint faculty positions at the Yale Law School and the Yale School of Management. But he's starting to establish a name in the business world as well by his promotion of Super Crunching, the use of randomized experiments and data mining predictive modeling techniques to elevate the performance of just about anything where results of alternative courses of action can be measured. Despite being relatively new for business, these methods have been used successfully to improve the effectiveness of government programs, educational initiatives, and healthcare delivery for many years, so there's a lot of precedent to guide the way.

Ian's thesis is pretty simple: Though “experts” bring a lot to their roles managing business, education, healthcare, sports teams, etc., there's a limit to such expertise that will generally cap performance – test scores in education, standard of living in government, profits in business, morbidity and mortality in healthcare, for example – without the assistance of Super Crunching. Experimentation with analytics can often predict and explain better than experts, and therein lies the rub. It's very difficult for many experts to acknowledge that their judgments can be enhanced by data, equations, and especially randomized experiments. A significant challenge, one recognized and addressed in Ian's work, is getting experts and Super Crunching to coexist peacefully – and optimally.

In the business world, Super Crunching is, of course, closely aligned with business intelligence (BI). Today's data warehouse provides the input to tomorrow's predictive models. For Ian Ayres, a passive data warehouse is a nice start, but not nearly enough. The maximum benefit of analytics will only be reached when predictive models are combined with randomized experiments – the full monty of Super Crunching – to determine cause and effect of strategic decisions. Want to design the ideal web experience for customers? Conduct tests. Wish to optimize customer marketing offers? Do randomized experiments or campaigns. Wish to design the next big product winner? Field test alternatives. Want to improve HR initiatives on employee satisfaction and longevity? Test alternative hypotheses with experiments. Want to optimize an investment portfolio to fund retirement? Consider an element of random portfolio allocation. Wish to learn what football plays are best in which situations? Call plays randomly.

The Ian Ayres' mantra for business intelligence: Don't simply measure performance, manage and enhance performance using the experimental tools of Super Crunching – prove that choices made are optimal. Going forward, I envision Super-Crunching business consultancies guiding their Fortune 1000 customers to new strategies drawn from experimentation and predictive modeling. I trust that BeyeNETWORK.com newsletter readers will find Ian's provocative ideas inspiring, providing new ways to think about their important BI initiatives. Keep up with Ian's work at http://www.supercrunchers.com. Enjoy.

QUESTION: You graduated summa cum laude from Yale and earned a Yale law degree followed by a Ph.D. in economics from MIT. You are now on the faculties of both the Yale Law School and Yale School of Management. How did all your education come together to get you to your positions today?

ANSWER: Well, I've always been interested in numbers and data. As an undergraduate economics major at Yale, I studied statistics and econometrics, and as a grad student in economics at MIT, I wrote a paper (that was later published) on the airline industry for a second year econometrics seminar. I think I really got into data crunching gear, though, after law school when I held a faculty position at Northwestern and was also on the staff at the American Bar Foundation. The Bar Foundation is a big proponent of statistical studies, and it was there that I first tested whether Chicago car dealerships discriminated on the basis of race and gender. I'm so passionate about the benefits of Super Crunching and its accessibility to even high schoolers, that I just jointly published an article in the Journal of the American Statistical Association with my pre-teenage son and daughter.

QUESTION: It seems every time I open a new Newsweek, Economist, BusinessWeek, or Wall Street Journal, I see a review of Super Crunchers. How would you characterize the reactions to the book? Have any surprised you? How has the academic world reacted?

ANSWER: I've been heartened by the reception of Super Crunchers, especially in the IT community. One employee of Offermatica, a company that specializes in randomized testing of web content, confided that his father finally understands the nature of his work now! And I don't believe an understanding of the benefits of Super Crunching is for geeks only. The book is not a heavy technical read, and the concepts are quite accessible to college graduates.

QUESTION: The main methodological weapons of Super Crunchers are predictive models, both statistical and machine learning, and randomized experiments. Your book differentiates from others on analytics in business by its emphasis – obsession – with randomization as a means of “proving” cause and effect of business strategies. Could you outline the major benefits of learning by experiments? What do you say to those who scoff at experiments as inappropriate or too messy or costly for business?

ANSWER: Predictive modeling, either statistical or machine learning, is the heart of Super Crunching. But you’re right, I am kind of obsessed with the power of randomized trials. With randomized experiments, businesses can test what causes what. By randomly assigning stores to two different groups, a business can powerfully estimate the impact of a policy.

And the use of randomized experiments in business is generally not as intrusive as often thought. If you’re a company with a substantial number of hits on your website, you’re making a big mistake if you’re not optimizing the elements of your site with randomized tests. It’s really cheap to run randomized tests on the web. And it’s not much harder to run randomized trials of mail-marketing or newspaper buys or even store planograms. In fact, experiments can be used to create better HR policies or test different inventory levels. Any business decision with repeated ongoing activities and a single metric of success may well benefit from the experimental method.
 
Along those lines, I’m working with companies now to use randomized tests to see if “commitment contracts” can help employees quit smoking and lose weight. But I also want to see if healthier employees have lower turnover and generate more sales. If any of your readers are interested in participating in this kind of study, they should feel free to e-mail me (or they can learn more at http://www.stickK.com).

QUESTION: In Super Crunchers, you offer a wealth of examples from not-for-profit sectors of the benefits of rigorous policy and program evaluation. Do you think business has historically been a laggard to government, education, and even healthcare in the assessment of its strategies and “programs”? If so, is the situation changing now?

ANSWER: Yes, surprisingly, business has been laggard in randomized testing. Perhaps that's because randomized testing is such a powerful bipartisan tool that it has a legacy with government programs, education, and healthcare. The randomized social experiment Progresa discussed in Super Crunchers, is a great example. The Progresa program gave poor mothers in Mexico cash and food supplements if their kids regularly went to health clinics and met school attendance goals. After a randomized test showed that Progresa kids were substantially healthier, the program was expanded nationally, and now 30 other countries have adopted Progresa-like subsidies. A single randomized test has had a humongous impact.

And business is randomizing now more than ever before. I'm currently consulting with a Fortune 50 company that is looking to expand its use of analytics and experimental methods first in e-commerce, then in its brick-and-mortar stores. Two other companies have asked for support in new Super Crunching training and auditing initiatives.

QUESTION: In a terrific Harvard Business Review interview, Jeff Bezos, CEO of Amazon, talks of the company's obsession with customers as driving from a meta strategy “to be stubborn on the vision but flexible on the details.” And for Amazon, the details often involve the execution of experiments, especially with the user interface on the website. “That is a huge laboratory for us, and we've put a lot of energy into trying to figure out how to be very low cost with those experiments so that we can run a much larger number of them.” Your thoughts?

ANSWER: Amazon is the prototype of a successful company driven by Super Crunching. Jeff Bezos is spot on with his emphasis of experiments on the Amazon web “laboratory.” As the cost of experiments decreases, you have the freedom to test a lot more hunches.

QUESTION: You collaborated on a very interesting study with Steve Levitt, co-author of widely acclaimed Freakonomics, on the benefits of the LoJack radio transmitter device in reducing the incidence of stolen vehicles. Levitt has become almost a rock star in the discipline of applying social science methods and statistics to real world decision-making problems. You also note the study suggesting point shaving in college basketball games by economist Justin Wolfers – work that is especially pertinent to the current situation in the NBA. And experimental economists like Esther Duflo design randomized trials to test practical strategies for combating poverty. It certainly appears that the scope of quantitative social science inquiry has broadened over the years, with now much more of an applied focus. What impact do you think this development will have on the future of Super Crunching? Do you see benefits of a quantitative social science background for business and government in contrast to pure statistics or machine learning? If you were a CEO, would you hire “freaks” to help steer your analytics initiatives?

ANSWER: Freakonomics author Steve Levitt is a hero whom I believe will someday win a Nobel prize for his “inventive empiricism” – creative ways to explain often mundane social phenomena using data. Steve, who has a part-time appointment at the American Bar Foundation in addition to his faculty position at the University of Chicago, can write for the masses as well as ratchet to a higher analytical gear for the academic world. “Freaks” like Steve have created a bit of a stir in academia by addressing different questions than the mainstream, but their value is now clearly visible to government, education, business – and universities. They are creating a new paradigm of economic analysis, one that includes fields like “forensic statistics” to help find the bad guys through the digital trails they often leave. I'd hire “freak” economists like Steve Levitt, Justin Wolfers and Esther Duflo in a heartbeat to Super Crunch in the business world. Their combinations of analytical prowess and business finesse would serve business well.

QUESTION: One of the recurring themes of Super Crunchers is the resistance from experts who are threatened by analytics. Yet, in case after case, the computer has demonstrated superiority as an analytical engine. Does a compromise between experts and analytics revolve on an expert focus to develop hypotheses and theories, with an analytics focus on testing same – and the collaboration of both to adapt better models?
 
ANSWER: Yes – that's where it's at: open ideas to testing, toggle between human intuition and computers, overcome resistance by adapting the best of both experts and statistics and incorporate expert opinion as a component of predictive models.

The more complicated the prediction – the more factors involved – the better Super Crunching performs relative to expert intuition. But there's even more to it than that. Super Crunching experiments are not one and done – experimentation must be ongoing to meet changing needs of business and programs. Also, companies should “diversify” their Super Crunching across multiple analysts to mitigate risk. They can then update or average the predictions across multiple sources, potentially gaining the “wisdom of the crowd.”

An interesting twist to the experts versus Super Crunching conundrum might be called “meta testing” to determine which types of experts serve the business best and in which situations. The executive team might be superior in several areas and front-line management in others. The good news is that their performance as experts can be evaluated.

QUESTION: Nassim Taleb's books Fooled by Randomness and The Black Swan put stakes in the ground for much less regularity and predictability in the business world (and life in general) than is generally imputed. Is this thinking at loggerheads with Super Crunchers? Or are we better to consider Taleb a scold and his tenets the null hypotheses that we are attempting to reject? Or other thoughts...?

ANSWER: Taleb is more of a scold, though he's right: many things defy prediction. Indeed, he makes a great point about prediction being about both estimates and precision. The problem with the big hedge fund declines over the summer may have been more about the lack of precision of predictions than the estimates themselves. Incidentally, a safeguard to the hedge fund problem might be having more, independent “quants” crunching data – providing checks and balances against the flawed genius in the corner.

The problem I have with both Fooled by Randomness and The Black Swan is that Taleb doesn’t give enough advice about how we should deal with a random, rare-event world. Investors still have to make decisions. You’re not going to put your money in a mattress. Crunching numbers to figure out the best way to diversify and to estimate the risk/return tradeoff is still the best way to go. It’s ironic that he’s made a lot of money focusing on statistical tendencies. Malcolm Gladwell has a great feature on him in the New Yorker explaining that Taleb built a successful hedge fund exploiting the fat tails of market returns.

QUESTION: It seems you're becoming a Super Crunching populist, making a host of prediction tools available on yourwebsite. Could you tell us a bit about it?

ANSWER: The prediction tools are intended to be both educational and fun. The individual models range from predicting the value of Bordeaux wines, pregnancy due dates, the results of sporting events, future heights of children, future world populations, lifetime income, the probability that your spouse will cheat on you, life expectancy, and the probability of heart disease. The 40 or so prediction tools hopefully give a sense of the scope of Super Crunching possibilities. We'll continue to expand the list over time.

QUESTION: Look into your crystal ball and give us a few predictions on the evolution of Super Crunching 5 to 10 years in the future. How will the policing of the Super Crunching “big brother” evolve as well?
 
ANSWER: Over the next 5 to 10 years, we should expect to see advances in medicine and healthcare, a direct consequence of a successful movement to digitize hospital records. As a result, there will be improvements in both diagnoses and treatment of disease.

The pendulum between Super Crunching and experts will swing several times. At first, there will be an over-reliance on number crunching – then a correction to expert predictions. In time, we will reach an expert/Super Crunching equilibrium, learning how best to toggle between experts and analytics. We will evolve our methods to optimally combine the predictions of multiple experts with multiple Super Crunchers.

My next book will be on how to better invest for retirement. The focus will be on stronger ways to diversify across assets and time. Using an extensive stock portfolio returns database, I will combine backtesting and Monte Carlo simulation methods to attempt to optimize asset allocation.

References:

1. Ian Ayres.Super Crunchers – Why Thinking-By-Numbers is the New Way to Be Smart. Bantam Books. 2007.
   
   
2. Steven D. Levitt and Stephen J. Dubner. Freakonomics – A Rogue Economist Explores the Hidden Side of Everything. HarperCollins. 2005.
   
   
3. Julia Kirby and Thomas A. Stewart. The Institutional Yes. Harvard Business Review. October 2007.

If you found this article helpful and would like to receive the latest insights each week from the experts featured on the BeyeNETWORK, please subscribe to the BeyeNETWORK Newsletter.