Is Agile BI Really a Better Mousetrap? Using Metrics to Validate Results

Originally published December 17, 2013

As my company began to evangelize and deploy Agile BI to our client base, we were constantly being asked the same question, “Do agile methods produce better results for business intelligence/data warehousing (BI/DW) implementations?” A fair question, I might add. In trying to figure this out, I researched the existing literature on the topic and came to realize that there was no quantifiable evidence that substantiated that agile methods led to better results. This led me to question the promise of Agile BI and what it was based on. Was it based on speculation? Platitudes? Empirical evidence?

Given the lack of published metrics, our BI practice leaders asked ourselves what we should measure beyond the standard burn-down and burn-up charts. While these two core metrics are useful to determine the overall health of a project while in flight, they don’t indicate whether traditional methods are better or worse.

To put agile metrics into perspective, then, we determined that we needed to measure a team’s:

  • Velocity: the rate at which work is completed

  • Efficiency: the cost to complete the work compared to the work effort (typically the number of deliverables and work products being produced1)

  • Quality: the number of inherent defects in the deliverables

Establishing a Baseline

To determine if Agile BI methods were working, we had to establish a baseline set of metrics and then compare the results within a release (from sprint to sprint) and across releases. We recorded each team’s performance, expecting that a team’s velocity and efficiency would increase while the defect rate decreased. To that end, we used PMI’s (Project Management Institute) concept of earned value and created the following metrics:
  • Cost Performance Index (CPI): Actual Hours / Planned Hours. This variation of the burn-down chart shows the percent of hours burned compared to the plan.

  • Results Performance Index (RPI): Actual Results / Planned Results. This permutation of the burn-up chart shows the percent of deliverables and work products completed compared to the plan.

  • Schedule Performance Index (SPI): Actual Project Days / Planned Project Days. This shows the percent of Project Days burned compared to Project Days planned.

  • Velocity Ratio = RPI/CPI. This shows the ratio of deliverables completed compared to the cost.

  • Efficiency Ratio = SPI/CPI. This ratio shows the schedule performance compared to the cost to achieve it.
From there, we created a dashboard that allowed us to measure the baseline metrics.2



Once the baseline metrics were established, we were in a position to measure Velocity and Efficiency across sprints and make adjustments as needed – if we weren’t seeing the expected improvements.



Using this same method, we could now compare projects and teams to help us understand the root cause of positive or negative trends and to identify best practices that could be adopted on a universal basis.

Another Basis of Comparison

While the above metrics were useful for helping us understand if velocity and efficiency were improving, we still couldn’t answer the question, “Do agile methods produce better results for BI/DW implementations?” In order to figure this out, we needed a basis of comparison. So we turned our attention to three different metrics:
  • Cost/Deliverable: The average cost to produce a deliverable

  • Defect Count: The number of inherent defects in the deliverables that are passed from the Scrum Team to the next step in the process (typically either the team responsible for System Test or for User Acceptance Test)

  • Break/Fix Costs: The costs to correct defects caused by the Agile/Scrum team
We segregated deliverables into distinct types such as:
  • ETL jobs3

  • Reports

  • Dashboards

  • Semantic layers

  • Database tables

Next, we began capturing hours for each team member working on a discrete deliverable so we could arrive at the metrics of interest. From there, it was simply a matter of comparing the metrics across sprints and across projects. The following charts show results for ETL jobs across sprints and shows how that translated into a productivity metric.4



While productivity gains are wonderful things, the next question we asked ourselves was, “Is quality suffering as a result of increased productivity?” To our delight, we discovered that it was not. Defects were trending the right way across sprints and as well as across releases.



These results helped us do two things:
  1. Validate whether or not we were seeing velocity and efficiency gains at the deliverable level without having to sacrifice quality

  2. Establish a baseline for comparing Agile BI costs to traditional delivery methods

Agile vs. Traditional Approaches

So here we were, congratulating ourselves on how well we were doing until we shared our results with one of our senior executives. His view was that the results looked great, but compared to what? This question led us to develop a survey that we could send to other BI shops whose information we could use to compare traditional development methods with the results we were seeing with agile. We did our best to create questions that would produce quantifiable and objective answers, and kept in mind that the concepts of Velocity and Efficiency wouldn’t necessarily be familiar to them. We took into account items such as team mix (internal resources vs. consultants and the rates associated with each) to arrive at a true cost. We made sure to include organizations that were using both local and offshore resources.

We then went about surveying organizations across a wide array of industries to produce three metrics to which we could compare our results:
  1. Initial Cost/Deliverable

  2. Break/Fix (coding defects and not enhancements)

  3. Developer Productivity
While we suspected that agile methods would produce better results, we did not expect them to be so dramatic. We found that the initial cost of developing ETL jobs using agile methods was, on average, 44.4% lower than those developed using a traditional delivery approach. Similarly, we found that break/fix costs for jobs created using traditional methods were, on average, 43.9 times greater than agile.5



On the productivity front, we found that ETL developers using agile methods were able to produce 350% more jobs than members of non-agile teams.



Better Results Confirmed

Having worked with Agile/Scrum methods for BI/DW projects for the past 4+ years, I firmly believe that they do lead to better results in terms of productivity, quality and cost, and we now have data to prove it. I also subscribe to the old adage that you can’t improve what you don’t measure. Our metrics support our decision to use Agile/Scrum, which has been further validated by our survey results.

For those of you who have begun applying Agile/Scrum, or are contemplating using them, I’d recommend the following:
  • Establish a baseline set of metrics to serve as a basis of comparison, making sure that definitions and calculations are documented, well-understood and applied consistently.

  • Measure your results to a fine grain such as at the deliverable level. We do this by recording results (planned vs. actual hours and/or story points) on the story cards themselves.

  • Use the results to work out the kinks if you’re not seeing improvements in velocity, efficiency and quality. We identify areas for improvement during the sprint retrospectives.

  • Understand that newly formed Agile/Scrum teams may not hit their targets in earlier sprints, but you should expect that their performance will improve during latter ones.

  • Use the metrics when planning the next sprint or estimating the next project. We’ve found that our metrics lead to better estimates.

  • Take the time to identify why some teams are performing better than others. These observations lead to best practices that can be applied in a universal manner.
Do agile methods produce better results for BI/DW implementations? We now have quantifiable evidence that prove they do. I’m a believer! How about you?

End Notes:
  1. Deliverables are defined as code (ETL programs, reports, dashboards, etc.) while work products are defined as items that support the deliverables (functional specifications, design specifications, etc.).
  2. We actually have two views – one for hours and one for story points.
  3. We have a working definition of what an” ETL job” means.
  4. The number of ETL jobs/day is a function of team size and will vary accordingly. The metric is intended to show whether or not team productivity is increasing, irrespective of size.
  5. We use test-driven development (TDD) methods.

  • Jim GalloJim Gallo
    Jim is  National Director, Vice President, Business Analytics at Information Control Corporation (ICC), a firm focused on reducing the cost of developing BI solutions. Jim and the ICC team have enabled companies to increase the velocity of their development by adopting Agile methods for BI, with a particular focus on turning theory into reality.

    Jim is a recognized expert and has published a number of articles on the practical realities of business intelligence and data warehousing. He is a regular speaker at conferences and industry-related events. He has led a number of large, complex BI projects for Fortune 1000 companies in addition to delivering value to federal and state governments and international clients. For two years in a row, Jim has been named IBM Champion for Information Integration and Federation. He can be reached at jgallo@iccohio.com.

    Editor's Note: More articles and resources are available in Jim's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

Recent articles by Jim Gallo

 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!