We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

The 12 (or so) Myths about Agile Business Intelligence – Myth 1 Agile BI is Radical and New

Originally published October 12, 2010

This article is the first in a series that will address the 12 myths about Agile BI that I listed on my BeyeNETWORK blog. The myth about Agile BI being radical and new, while being fairly common, is also one of my favorites. But first, a couple of stories from the trenches.

Story One begins with a meeting with a project management office (PMO). As I sit in their “war room” I’m explaining the methods and benefits of Agile BI delivery – iterative, business-driven, delivery focused, scalable and rapid. I am explaining how to break down business intelligence (BI)/data warehousing (DW) projects into ten to twelve week delivery cycles that allow the business to gain value in small, manageable steps instead of having to wait six to twelve months for the first implementation (also known as the “Big Bang” approach). I go on to explain how Sprint cycles work, each cycle lasting around two weeks, resulting in potentially deployable code. While this is going on, I notice a 30-foot wall with a 12-month project plan that would, at least in theory, deliver the BI/DW solution. After explaining my approach, the head of the PMO and the lead project manager tell me that what I am proposing is “too radical” because “the business doesn’t want to move that fast” and “they are comfortable with six to twelve month delivery cycles.” Say what?

Story Two involves a similar discussion, this time with a combined audience of business and IT staff. I explain how an Agile BI approach starts with the Business Value Chain, with requirements being defined first in generic terms such as “Improve Customer Retention” (epics and themes in the Agile vernacular) and then decomposed into questions and goals (user stories). I go on to explain how questions then get decomposed into business terms from which a business glossary (dictionary) is created. From there, the team then identifies candidate source systems from which the warehouse will be populated (technical stories). I speak about the iterative approach, with requirements, design, prototyping, development and testing all being done multiple times within each Sprint cycle so that at the end of the Sprint something of value gets created. After the explanation, the business folks get very excited and want to move forward. My IT brethren react like horses rearing up when coming across a rattlesnake. They explain to me and their business constituents that they have blocked out six months for requirements definition and would move onto design and development only after requirements have been defined. My response is that I can have a deployable solution done in less time than they’ve allocated for requirements, but IT is still hesitant to move forward. Say what?

What do these stories have in common? First, irrespective of Agile BI methods, there’s a whole lot of old-school thinking out there, as these types of discussions are, unfortunately, not uncommon. These IT groups appear to be very much entrenched in the traditional waterfall methodology that purportedly mandates you can’t proceed from one SDLC step until you’ve completed the previous step. I’m pretty sure this type of reaction has less to do with Agile BI and more to do with a discomfort around a different way of delivering BI solutions. Reading between the lines, what I think these groups were really saying is, “We don’t want to change the way things are done. We’re comfortable with how things get done and if we adopt a new way of delivering BI solutions, it means more work for IT, regardless of the advantages for the business.”

More importantly, take a look at the highlighted words in the first two paragraphs. What do you see? Things like “iterative,” “business-driven,” “business glossary,” “requirements,” “prototyping,” etc. From my perspective, none of these things are radical or new and are not necessarily the brainchild of the Agile school. I’m pretty sure that many organizations take a business-driven and iterative approach to delivery. I’m absolutely certain that one must understand requirements before designing something, and that data stewardship (business terms/glossary) is a key underpinning of any BI/DW project.

Quite honestly, I don’t remember when or how I leaned that iterative development was a key tenet of BI/DW delivery and that BI projects must start with a business need and that the business must be active participants throughout the delivery process. Like many of you, I read the classics like Corporate Information Factory  by Bill Inmon, Claudia Imhoff and Ryan Sousa; the Data Warehouse Lifecycle Toolkit  by Ralph Kimball, et al; and Business Intelligence Roadmap  by Larissa Moss and Shaku Atre. In them, the authors speak quite clearly about taking an iterative approach to delivery, the importance of business participation and how prototyping is an important step in the process. Oh, and by the way, these books were published 8-10 years ago!

Now, let’s go back to the two stories. In both cases, the project managers and IT groups were following the "big bang” approach. This approach has been debunked so often I really don’t know who to give credit to. I do remember reading a number of articles and bi-folds on “The 10 Reasons Data Warehouse/BI Projects Fail” and other similar publications from organizations such as TDWI. More times than not, “big bang,” lack of business involvement, lack of data standards and data-driven approaches made everyone’s top 10 list. Using the business requirements to drive candidate source system selection? Not new and certainly not radical. Creating prototypes to gain clarity and expanding the project outward from there (aka spiral)? Not new!

So what is radical and new? From my perspective, nothing about Agile BI is radical. Once you get comfortable with it, you will quickly conclude that it’s mostly common sense. What’s new is applying Agile/Scrum methods to BI projects.

The following diagram provides a context for the Agile BI delivery process that I helped architect for my own company, Information Control Corporation.

Figure 1:
ICC’s Agile BI Process
(mouse over image to enlarge)

One key to Agile BI is the process one follows to decompose the BI solution into small, manageable pieces that get articulated as story cards. Story cards are then allocated to a two-week “mini-SDLC” referred to as a Sprint. Once a Sprint begins, each card is meted out to the project team, with the product backlog serving as the scope and the story cards themselves serving as the task list. The resulting artifacts are assembled to produce deliverables. This decomposition and reassembly process is shown in the following diagram, using a BI project focused on customer metrics as an example.

Figure 2: Decomposition and Assembly
(mouse over image to enlarge)

Another key of applying this methodology is to organize the work so that at the end of every two weeks, the team produces “potentially deployable ‘code’.” The following diagram puts the Value Chain processes in the context of the overall BI/DW SDLC, as this is key to understanding the use of the term “potentially.”

Figure 3: ICC’s Agile BI Framework within the Entire SDLC
(mouse over image to enlarge)

Note that Steps 1-3, 15 and 16 are based on the delivery methodology as defined in Business Intelligence Roadmap. What we have done is incorporate product backlog tasks in the Justification and Initiate stages and replaced the Business Analysis, Design and Construction stages with the Business and Technical Value Chains. I’ll be writing more about this as I discuss myths 3 and 4 – “Agile BI methods do not support sound BI/DW architecture principles” and “Agile BI displaces the software development lifecycle (SDLC) in its entirety.”

It’s essential to understand that just because some artifacts may have been produced and are production-ready (Sprint Release), it doesn’t necessarily mean you deploy them. For example, the epic and themes may be focused on customer analytics (profitability, retention, etc.). At the end of the first Sprint, you may have produced a working prototype of the BI front-end with a single fact table and a few key dimension tables created. Is it deployable? Well, the tables certainly may be, but the working prototype may not have been hardened enough to put into production. Similarly, the single fact table and the dimension tables may not yet provide enough information to be of significant use to the business. It might make more sense to wait for two or three Sprints’ worth of work to be ready before the solution gets deployed. More commonly, we must also abide by a rigorous promotion process (promoting the solution from Test/QA to Production), and the timing may be off. For example, many organizations only deploy solutions into production once a quarter or may require the team to have everything ready for production two to three weeks ahead of time so that it can be scheduled for production. Therefore, while we manage and organize around Sprints, we must also plan Sprint releases around iterations and configuration management processes.

In a nutshell, Agile BI is a variation of iterative delivery. It embraces best practices established long ago for BI/DW projects and weds those best practices to the best that the Agile school has given us. I hope I’ve shown you that Agile BI isn’t new, and I’d love to hear from anyone who believes that it is radical. Stay tuned as I debunk Myth 2 – “There is no need to produce documentation when using Agile BI methods” – as we head down "The Path to Agility."

  • Jim GalloJim Gallo
    Jim is  National Director, Vice President, Business Analytics at Information Control Corporation (ICC), a firm focused on reducing the cost of developing BI solutions. Jim and the ICC team have enabled companies to increase the velocity of their development by adopting Agile methods for BI, with a particular focus on turning theory into reality.

    Jim is a recognized expert and has published a number of articles on the practical realities of business intelligence and data warehousing. He is a regular speaker at conferences and industry-related events. He has led a number of large, complex BI projects for Fortune 1000 companies in addition to delivering value to federal and state governments and international clients. For two years in a row, Jim has been named IBM Champion for Information Integration and Federation. He can be reached at jgallo@iccohio.com.

    Editor's Note: More articles and resources are available in Jim's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

Recent articles by Jim Gallo



Want to post a comment? Login or become a member today!

Posted December 7, 2010 by Leslie Ament lar@hypatiaresearch.com


It is so refreshing to have someone share a reality-check assessment of how and why agile BI is on a slow adoption cycle.  Closed loop iterative methodologies & techniques typically yield higher ROIs for clients.....keep educating them!   @Hypatia_18

Is this comment inappropriate? Click here to flag this comment.