Business Intelligence Network - The Vision for BI and Beyond

Strategic Learning

Once a year I pull out my staple of articles and books by balanced scorecard (BS) authors Robert Kaplan and David Norton. It seems this small effort always more than pays for itself in providing insights on the nature of business intelligence. Indeed, I’ve probably progressed to where I’d rate the balanced scorecard as among the most important conceptual paradigms for business intelligence (BI) today.

Balanced scorecard is a system for managing the development, execution, evaluation and improvement of a company’s strategy and processes. The closed-loop stages outlined by the authors in a recent Harvard Business Review article, "Mastering the Management System" –

1. Develop the Strategy,
2. Translate the Strategy,
3. Plan Operations,
4. Monitor and Learn, and
5. Test and Adapt the Strategy

 – comprise a simple yet intuitive and methodical approach to management that resonates well with both business and intelligence professionals.¹

The balanced scorecard addresses three areas critical to strategic learning. First, it details a vision, a clear articulation of what the company is attempting to achieve. Second, it provides the overall strategic feedback system, identifying how the company will attain its vision. And third, the scorecard promotes feedback and review that is sine qua non for closing the loop between vision and results – providing the basis for strategic learning. “A business strategy can be viewed as a set of hypotheses about cause-and-effect relationships. A strategic feedback system should be able to test, validate and modify the hypotheses embedded in a business unit’s strategy.”²

Kaplan and Norton’s focus on strategy as a mosaic of linkage hypotheses – a strategy map – is an ideal point of departure for business intelligence (BI). These linkages detail relationships between exogenous factors at the company’s control (X), leading performance indicators (Y) and lagging indicators (Z). Strategy is thus represented as a series of statements of the form: “If we do X, then Y will increase, leading to an enhanced financial indicator Z.” Alternatively, the business could specify: “The more we do of X, the lower will be measure Y, which, in turn, will lead to better financial performance Z.” The challenge for strategy is to establish X, Y and Z in a chain of cause and effect relationships where X causes Y which, in turn, leads to Z. A primary function of business intelligence is then to test these linkages to evaluate the effectiveness of strategy and offer recommendations for change.

An illustration of this approach offered by the authors is as follows: Recognizing the need to satisfy customer expectations for on-time delivery (Y2), a business identified several internal processes – order processing, scheduling, and fulfillment – in which it should excel (Y1). To differentiate with these processes, the company chose to invest in employee training and new information systems (X). The resulting internal improvements would lead to an increase in margins (Z). In sum, a company investment in systems and training (X) would lead to improved internal processes (Y1), which would, in turn, enhance customer satisfaction with fulfillment (Y2), leading ultimately to increased margins (Z).

Evaluation Designs for Business Intelligence

Ideally, strategists and BI analysts are closely aligned as they collaborate to deploy a management system based on the strategy as hypotheses foundation for learning. Strategists assume the role of corporate theorist, while BI plays research scientist. Strategists develop the causal linkages; BI tests them. BI analysts work closely with the strategy team, helping operationalize the linkage hypotheses, proposing performance measures and analytics. BI is then tasked with executing the evaluation solution. The two work jointly to improve future hypotheses with feedback from business intelligence findings. 

BI analysts use data and analytics to test assertions that factor X causes indicator Y. With the role of research scientist, however, the analysts cannot be content to simply show a correlation between X and Y, but must also prove that X causes Y, by eliminating alternative hypotheses that other factors, like X2 and X3, are the real causes of Y. BI must, therefore, be especially attentive to the designs they deploy to gather intelligence. 

Effective design is a critical, but often underappreciated, aspect of the BI evaluation process for testing components of the strategy linking company initiatives to performance. The tighter these designs, the more assurance the company can have of its BI findings. The gold standard for establishing the validity of strategic hypotheses is, of course, the randomized experiment. Randomization to treatment helps assure that observed differences in performance variables between experimental and control groups are due to the intervention, and not to other uncontrolled factors that might be related to the performance measures and, subsequently, be sources of bias. With randomization, those bias-causing, uncontrolled factors should, on the average, be equal between intervention and control groups.

At a minimum, BI practitioners should understand the strengths and weaknesses of the designs they deploy to gather intelligence so they know how much confidence to place in their BI findings. Consider the seven simple designs often used for business intelligence outlined in Table 1, where O represents observation or measurement, X is a strategic intervention, and R denotes randomization.³ Design 1a, the one-shot case study, offers no possibilities to learn from comparisons or over-time contrasts and is really not much of a design at all.  Yet this “design” is quite pervasive in business intelligence, underpinning much of predictive modeling, and is often a significant foundation for findings that impact business decision making. For the linkage example articulated above, design 1a would not observe baseline measures of strategic variables, noting correlations of the post variables only. The business team would, therefore, be unable to note changes in the leading and lagging indicators (Y1,Y2, Z) and could draw no conclusions about the effectiveness of the strategic intervention (X). The one group pretest-posttest 1b introduces at least a simple pre-post comparison of the investigation units, providing a baseline to assess change. The main problem with 1b is that differences in the pre and post measurements might be due to factors, such as history, other than the strategic intervention – and this design is hard pressed to refute such alternative explanations.

Both pure experimental designs 2a and 2b should be standards by which BI aspires to gather intelligence. The power of randomization of units to either intervention or control groups, along with the benefits of pre and post measurements, make these simple designs well able to withstand threats to the validity of inquiries. And, in the Internet age, it’s often pretty straightforward to execute simple randomized experiments that can assure the quality of results. Companies like Google, Capital One, Amazon, Yahoo and Harrah’s routinely base strategic decisions on the findings of randomized experiments. In some cases, savvy companies make a series of low risk “flier” experiments, searching for serendipitous positive results. Capital One, for example, uses the power of randomized experiments to help optimize its credit card mix of interest rates, annual fees and awards, often with unintuitive findings.

For those instances in which randomization is impractical or inappropriate, quasi-experimental designs 3, 4a, and 4b, supplemented by statistical adjustments for bias, might be acceptable substitutes. Design 3 introduces a next level of complexity to pre-experiments by adding a comparison or control group to the analysis. Indeed, design 3 looks much like its pure experimental cousins, except that it uses “natural” groups instead of randomization to intervention/control. Without the benefits of randomization, selection and other biases can distort findings – misleading analysts to conclude there are differences between intervention and control, when in fact the groups are different (there are biases) out of the gate. In our example, the new strategy (X) might be applied to a single region, with the remaining regions serving as controls. This design has the benefits of both pre and post measurements as well as treatment/control groups. Unlike with random assignment where treatment and control are “equal,” however, it may be the case that the regions themselves and not the strategic interventions are responsible for measurement differences. Statisticians at times use sophisticated matching techniques to equate treatment and control groups where randomization is impractical, and deploy propensity models to adjust findings for initial differences in treatment and control. These adjustments are often successful in reducing the bias of the non-randomized investigations.

Designs 4a and 4b are extensions of 1b and 3, respectively, and are probably quite accessible to the BI world today. The information from the additional observations both pre and post strategic intervention might play a significant role in assuring the validity of performance findings. With 4a, for example, if measurements taken before the intervention have similar values, and those taken after are similar to each other but greater than the before, an argument can be made for a treatment effect. On the other hand, if all observations both pre and post intervention lie on a straight line over time, analysts would be hard pressed to claim an impact of the intervention.

Design 4b is particularly strong, with the treatment/control groups and the multiple observations pre and post intervention collaborating to enhance the validity of findings. Though not quite as powerful as a randomized experiment, this design does a credible job controlling the significant biases of selection and history. The design is also unobtrusive, feasible to implement in many BI situations. Along with randomized experiments, multiple time series should make the short list of designs deployed to assess the performance of strategy.

Table Key:
O =observation or measurement
X = strategic intervention
R = randomization

Conclusion

The metaphor of strategy as hypotheses provides a powerful framework for developing, understanding, evaluating and improving business strategy. If strategists and the senior management team are the theoreticians, then BI analysts are the research scientists that use data, designs and analytics to test and help improve business performance. This amalgam of theory and testing provides a compelling foundation for corporate strategic learning.

References

1. Robert S. Kaplan and David P. Norton. “Mastering the Management System”. Harvard Business Review. January, 2008.
2. Robert S. Kaplan and David P. Norton. “Using the Balanced Scorecard as a Strategic Management System.” Harvard Business Review on Measuring Corporate Performance. Harvard Business School Publishing Corporation. 1998.
3. Donald T. Campbell and Julian C. Stanley. Experimental and Quasi-Experimental Designs for Research. Rand McNally College Publishing. 1963.

If you found this article helpful and would like to receive the latest insights each week from the experts featured on the Business Intelligence Network, please subscribe to the Business Intelligence Network Newsletter.