Is Agile BI Really a Better Mousetrap? Using Metrics to Validate Results
by Jim Gallo
Originally published December 17, 2013
As my company began to evangelize and deploy Agile BI to our client base, we were constantly being asked the same question, “Do agile methods produce better results for business intelligence/data warehousing (BI/DW) implementations?” A fair question, I might add. In trying to figure this out, I researched the existing literature on the topic and came to realize that there was no quantifiable evidence that substantiated that agile methods led to better results. This led me to question the promise of Agile BI and what it was based on. Was it based on speculation? Platitudes? Empirical evidence?
Establishing a BaselineTo determine if Agile BI methods were working, we had to establish a baseline set of metrics and then compare the results within a release (from sprint to sprint) and across releases. We recorded each team’s performance, expecting that a team’s velocity and efficiency would increase while the defect rate decreased. To that end, we used PMI’s (Project Management Institute) concept of earned value and created the following metrics:
Once the baseline metrics were established, we were in a position to measure Velocity and Efficiency across sprints and make adjustments as needed – if we weren’t seeing the expected improvements.
Using this same method, we could now compare projects and teams to help us understand the root cause of positive or negative trends and to identify best practices that could be adopted on a universal basis.
Another Basis of ComparisonWhile the above metrics were useful for helping us understand if velocity and efficiency were improving, we still couldn’t answer the question, “Do agile methods produce better results for BI/DW implementations?” In order to figure this out, we needed a basis of comparison. So we turned our attention to three different metrics:
Next, we began capturing hours for each team member working on a discrete deliverable so we could arrive at the metrics of interest. From there, it was simply a matter of comparing the metrics across sprints and across projects. The following charts show results for ETL jobs across sprints and shows how that translated into a productivity metric.4
While productivity gains are wonderful things, the next question we asked ourselves was, “Is quality suffering as a result of increased productivity?” To our delight, we discovered that it was not. Defects were trending the right way across sprints and as well as across releases.
These results helped us do two things:
Agile vs. Traditional ApproachesSo here we were, congratulating ourselves on how well we were doing until we shared our results with one of our senior executives. His view was that the results looked great, but compared to what? This question led us to develop a survey that we could send to other BI shops whose information we could use to compare traditional development methods with the results we were seeing with agile. We did our best to create questions that would produce quantifiable and objective answers, and kept in mind that the concepts of Velocity and Efficiency wouldn’t necessarily be familiar to them. We took into account items such as team mix (internal resources vs. consultants and the rates associated with each) to arrive at a true cost. We made sure to include organizations that were using both local and offshore resources.
We then went about surveying organizations across a wide array of industries to produce three metrics to which we could compare our results:
On the productivity front, we found that ETL developers using agile methods were able to produce 350% more jobs than members of non-agile teams.
Better Results ConfirmedHaving worked with Agile/Scrum methods for BI/DW projects for the past 4+ years, I firmly believe that they do lead to better results in terms of productivity, quality and cost, and we now have data to prove it. I also subscribe to the old adage that you can’t improve what you don’t measure. Our metrics support our decision to use Agile/Scrum, which has been further validated by our survey results.
For those of you who have begun applying Agile/Scrum, or are contemplating using them, I’d recommend the following:
Recent articles by Jim Gallo
Copyright 2004 — 2019. Powell Media, LLC. All rights reserved.
BeyeNETWORK™ is a trademark of Powell Media, LLC