Business Intelligence Network - The Vision for BI and Beyond

Jeff Pfeffer and Bob Sutton are busy guys. In addition to their distinguished faculty positions at Stanford University, Jeff and Bob have been publishing books like there’s no tomorrow. And that’s very good for a business reading market glutted with feel-good, but often dangerous, theme-of-the-year best-sellers.

I was first exposed to the writing of Pfeffer and Sutton courtesy of the recommendation engine of Amazon, which suggested I might like their 2000 book, The Knowing-Doing Gap: How Smart Companies Turn Knowledge into Action. I did indeed like the book. The central theme of The Knowing-Doing Gap is the irony that companies that appear to have substantial insight into how to improve their businesses often fail to transform that knowledge into action. Pfeffer and Sutton clearly articulate this knowing-doing gap and offer sensible prescriptions on how to put knowledge to work for business gain.

Amazon’s collaborative filtering engine was at it again early last year, suggesting I try Pfeffer and Sutton’s then new book: Hard Facts, Dangerous Half-Truths & Total Nonsense: Profiting From Evidence-Based Management. In Hard Facts, Pfeffer and Sutton build on the evidence-based paradigms of medicine, education and government to promote evidence-based management (EBM). Focusing on poor management decision practices like casual benchmarking of others’ successful experiences, a preoccupation of what worked in the past, and deeply held yet unsupported management convictions, Pfeffer and Sutton offer an evidence-based antidote that derives from data and facts, from skepticism of the next big approach, and from wisdom built on the constant evaluation of knowledge and the humility to doubt and change. Business intelligence is no doubt closely aligned with EBM, providing a basis for facts and the means to evaluate and change courses of action.

To say I caught up with Jeff Pfeffer and Bob Sutton is a simplification. After early discourse, Bob was out promoting yet another book, The No Asshole Rule: Building a Civilized Workplace and Surviving One That Isn’t. (I can imagine that being a lengthy trip.) I did, however, manage to connect with them on several occasions in late August and early September, and I am glad I did. Jeff and Bob generously gave their time when I’m sure it was least available. For that collaboration, I am most grateful. I hope that Business Intelligence Network (BeyeNETWORK.com) readers enjoy their wisdom. 

QUESTION: You are both professors at Stanford University – Jeff from the Graduate School of Business and Bob from the Technology Ventures Program. Could you give us some additional background information?

ANSWER: Bob is actually a professor in the Management Science and Engineering department in the School of Engineering. He received his Ph.D. in organizational psychology from the University of Michigan and has taught at Stanford since 1983. In addition to being active in the new Design School, Sutton has served as co-director of the Center for Work, Technology, and Organizations. Pfeffer, who got his Ph.D. from Stanford, has been on the faculty since 1979, having previously taught at U.C. Berkeley and the University of Illinois. He has been a visiting faculty member at Harvard Business School, Singapore Management University, London Business School, and IESE in Barcelona.

QUESTION: Your 2006 Harvard Business School Press book, Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-Based Management, has been widely acclaimed. Could you tell us how you came to the understanding that principles similar to those derived from evidence-based medicine and education are pertinent for business? How pervasive do you think EBM is in business today?

ANSWER: To answer the second question first, EBM is not very pervasive. Many decisions in business are based on casual benchmarking – watching what other companies are doing – on people’s interpretation of their past experience, on what executives seem to be good at doing, at what is being hyped in the business press – in short, on many things other than theory and facts. We saw that a lot of people were doing things – trying to turn knowledge into action – but what they were doing was inconsistent with the theory and evidence from much of psychology and organizational behavior. When one begins to consider how to make better decisions, the evidence-based movement, not just in medicine but in education, criminology, and other policy sciences, just seems like a sensible place to start. Looking at those movements reveals similarities with the issues faced in management, and also highlights both the challenges faced and some possible ways of proceeding to meet those challenges.

QUESTION: Evidence-based medicine has been characterized by David Sackett of Oxford as “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients…integrating individual clinical expertise with the best available clinical evidence from systematic research.” EBM proponents espouse an “evidence hierarchy” that ranks studies and methods on research rigor – from the low-end of anecdotal evidence to the gold standard of randomized controlled trials. How pertinent is the evidence hierarchy and methodological rigor for bringing the best available evidence to develop/test business hypotheses? Could you site examples of companies adopting high-quality research designs – either experimental or quasi-experimental – to determine optimal strategies? Is such research on the rise?

ANSWER: Methodological rigor is important, but diagnostic thinking and sound analysis may be even more important. As Gary Loveman of Harrah’s Entertainment said to a Stanford class, what he tries to teach Harrah’s people is how to ask good questions, for instance: Why are customers who could be choosing Harrah’s choosing a competitor instead? Is it because Harrah’s isn’t offering the right things, or that the pricing is wrong, or that people don’t see what the offerings are, and so forth? He believes that if people learn how to ask smart questions, they will figure out how to get good answers. By the way, this principle is true in research in general. Rigorous methods applied to an unimportant or poorly framed question won’t produce much. So, the key issue is to learn how to ask good questions. Harrah’s, Yahoo, the Home Shopping Network, Capital One, and various retailers and financial institutions have been using evidence-based experiments for a while. It is hard to know how much this practice is growing.

QUESTION: In addition to methodological rigor for testing, EBM prescribes a strong evaluation/feedback loop to gauge the efficacy of courses of action, cataloging such information for subsequent iterations. Does EBM argue for more of a bottom-up, learn as you go, unfinished prototype development of strategy? Are there areas of business where a “learn as you go” strategy is superior? Are there areas where traditional top-down strategy should be the choice?

ANSWER: We don’t think this is either-or. Even in medicine, with its double-blind, placebo controlled trials, learning from clinical practice in a systematic fashion is important. It is obviously crucial to admit mistakes and do follow up to see what works, and the rapid-prototyping of design thinking has much to recommend it. But learning as you go as well as learning “before you go” are both important dimensions of evidence-based management.

QUESTION: One of the psychological tenets of EBM is what you call “the attitude of wisdom” – acting with knowledge (while doubting what you know). In her terrific book, The March of Folly, Barbara Tubman defines “wooden-headedness” as a psychological malady that manifests in “assessing a situation in terms of preconceived fixed notions while ignoring or rejecting contrary signs.” How important is psychological disposition, especially absence of leadership ego, to the success of EBM initiatives?

ANSWER: Very important. It is hard to practice evidence-based management effectively in an organization with a culture that does not support it. Too many leaders overestimate their importance and people in general have excessive levels of confidence about events they have predicted or helped to create. As one famous quotation goes, what gets you into trouble isn’t what you don’t know, it’s what you think you know that isn’t so. If you are uncertain, you will seek out help and information. But if you think you know something, you will act on the basis of that knowledge even if mistaken, and often will persist even in the face of evidence that you are mistaken. In fact, as the literature on cognitive dissonance illustrates, often the more evidence disconfirms previously held beliefs, the stronger those beliefs are embraced by people who are committed to them. Thus, the most important leadership quality is a willingness to admit what you don’t know or aren’t certain about, and a willingness to change your mind when confronted with different evidence and ideas. It is a quality of intellectual openness – and it is at the foundation of an attitude of wisdom. Unfortunately, few of the selection practices or ways in which people rise up in hierarchies – either in companies or in the political realm – seem to select for this quality.

QUESTION: An article in the business section of the June 5, 2007, Chicago Tribune entitled Drugs in comeback against artery stents summarizes several issues surrounding current treatments for coronary artery disease. Citing both a controversial study that argues stents and angioplasties are no more effective in preventing heart attacks than drug therapy, and the documented blood-clotting side effects of some drug-coated stents, the article notes the “conflicts” of physicians who must sort through the welter of evidence as advocates for their patients. “If you don’t question what you do on a daily basis, you are not being responsible; so, appropriately, there is always a reevaluation and rethinking,” said Dr. Peter Kerwin, medical director of the cardiac catheterization lab at Advocate Good Samaritan Hospital in Downers Grove, Illinois. Is Dr. Kerwin practicing good EBM behavior?

ANSWER: His quote summarizes the essence of good EBM behavior. And, in fact, his quote summarizes what smart leaders who want to learn do.

QUESTION: Business intelligence (BI) can be defined as the use of data, technology, methods, and analytics to measure and improve the performance of business processes. It would seem that a well-functioning BI focus is sine qua non for EBM. Your thoughts?

ANSWER: BI is necessary but not sufficient. As a market researcher remarked a while ago, the problem with companies isn’t that they don’t have the data or the analytics, or even the studies – it’s that invariably they don’t use this information when they actually make decisions. That’s why our book emphasizes so strongly the mind-set or way of thinking that forms the foundation of evidence-based management.

QUESTION: An argument might be made that the application of evidence-based approaches popularized in medicine and education to business is to some extent inappropriate. While solid EB methods might result in practices that lift all patients or all students, such outcomes are impossible for business since businesses compete where performance is relative – there are always winners and losers. To use an investment analogy, the sum of all portfolio performance is the market. Your reactions?

ANSWER: We don’t agree at all. First of all, no drug or surgical procedure works on everyone. No public policy intervention is 100% effective. Every intervention has costs and benefits, and reactions and outcomes vary. So, people need to weigh costs and benefits and not have unrealistic expectations for what evidence can do, in any domain. Second, every management intervention and idea – ranging from competitive strategy analysis to the use of operations research to optimize refinery performance – can be, and eventually is, copied. That just raises the bar for further performance improvements. But we believe, and some others agree, that in an increasingly competitive world, not making the best decisions possible by asking smart questions and then figuring out how to answer them will leave companies struggling.

QUESTION: In Hard Facts, you note the inefficiencies of the business ideas market, in one table illustrating no less than nine diametrically opposed “formulas” for business success. Phil Rosenzweig has categorized such guru “delusions” in his outstanding book: The Halo Effect. Nassim Nicholas Taleb, in his ruminations on randomness and improbable events, also laments such delusional business thinking. Will the accumulated “evidence” against such flawed formulaic thinking turn against popular good-story business books – and in favor of less sexy EBM?

ANSWER: Not for a while. The problem, by the way, isn’t with the authors – it’s with the market. Business authors, and business journalists, are just supplying what the market seems to want. Until leaders become more sophisticated consumers of business advice, and that includes the advice delivered by management consulting firms, not much is going to change.

QUESTION: What do you see as the short-term and long-term prognosis for EBM? Will EBM become standard business practice in our lifetimes?

ANSWER: That depends on how much longer we will live. I think we need to be both committed and realistic. We have started a website, to gather information and resources to try to move the evidence-based management movement further along. There are efforts underway in our professional association, The Academy of Management, to promulgate evidence-based management. EBM is important, particularly given the harmful effects we see of so many common and popular management practices that aren’t based on the evidence. There is certainly increased interest in EBM. On the other hand, the evidence-based medicine movement has been at it a long time, and the movement of the best evidence and standards of practice from the laboratory to the bedside is still quite slow and sporadic. At the Mayo Clinic, they claim that they get new standards of practice implemented in 2-3 years – way better than the 15 that is more typical for physicians in general but still, in their opinion, too long for important knowledge to diffuse. The likelihood is that EBM will find its first foothold in healthcare administration. That’s because doctors are asking, “Why do we have to use evidence and you administrators don’t.” Healthcare is an enormous sector of the economy, and it is likely that EBM will diffuse from that sector into other places as well. But while resistance may be futile in the long run, EBM has and will run into some determined resistance.