This article is the first in a series that will address the 12 myths about Agile BI that I listed on my BeyeNETWORK blog. The myth about Agile BI being radical and new, while being fairly common, is also one of my favorites. But first, a couple of stories from the trenches.

Story One begins with a meeting with a project management office (PMO). As I sit in their “war room” I’m explaining the methods and benefits of Agile BI delivery – iterative, business-driven, delivery focused, scalable and rapid. I am explaining how to break down business intelligence (BI)/data warehousing (DW) projects into ten to twelve week delivery cycles that allow the business to gain value in small, manageable steps instead of having to wait six to twelve months for the first implementation (also known as the “Big Bang” approach). I go on to explain how Sprint cycles work, each cycle lasting around two weeks, resulting in potentially deployable code. While this is going on, I notice a 30-foot wall with a 12-month project plan that would, at least in theory, deliver the BI/DW solution. After explaining my approach, the head of the PMO and the lead project manager tell me that what I am proposing is “too radical” because “the business doesn’t want to move that fast” and “they are comfortable with six to twelve month delivery cycles.” Say what?

Story Two involves a similar discussion, this time with a combined audience of business and IT staff. I explain how an Agile BI approach starts with the Business Value Chain, with requirements being defined first in generic terms such as “Improve Customer Retention” (epics and themes in the Agile vernacular) and then decomposed into questions and goals (user stories). I go on to explain how questions then get decomposed into business terms from which a business glossary (dictionary) is created. From there, the team then identifies candidate source systems from which the warehouse will be populated (technical stories). I speak about the iterative approach, with requirements, design, prototyping, development and testing all being done multiple times within each Sprint cycle so that at the end of the Sprint something of value gets created. After the explanation, the business folks get very excited and want to move forward. My IT brethren react like horses rearing up when coming across a rattlesnake. They explain to me and their business constituents that they have blocked out six months for requirements definition and would move onto design and development only after requirements have been defined. My response is that I can have a deployable solution done in less time than they’ve allocated for requirements, but IT is still hesitant to move forward. Say what?

What do these stories have in common? First, irrespective of Agile BI methods, there’s a whole lot of old-school thinking out there, as these types of discussions are, unfortunately, not uncommon. These IT groups appear to be very much entrenched in the traditional waterfall methodology that purportedly mandates you can’t proceed from one SDLC step until you’ve completed the previous step. I’m pretty sure this type of reaction has less to do with Agile BI and more to do with a discomfort around a different way of delivering BI solutions. Reading between the lines, what I think these groups were really saying is, “We don’t want to change the way things are done. We’re comfortable with how things get done and if we adopt a new way of delivering BI solutions, it means more work for IT, regardless of the advantages for the business.”

More importantly, take a look at the highlighted words in the first two paragraphs. What do you see? Things like “iterative,” “business-driven,” “business glossary,” “requirements,” “prototyping,” etc. From my perspective, none of these things are radical or new and are not necessarily the brainchild of the Agile school. I’m pretty sure that many organizations take a business-driven and iterative approach to delivery. I’m absolutely certain that one must understand requirements before designing something, and that data stewardship (business terms/glossary) is a key underpinning of any BI/DW project.

Quite honestly, I don’t remember when or how I leaned that iterative development was a key tenet of BI/DW delivery and that BI projects must start with a business need and that the business must be active participants throughout the delivery process. Like many of you, I read the classics like Corporate Information Factory  by Bill Inmon, Claudia Imhoff and Ryan Sousa; the Data Warehouse Lifecycle Toolkit  by Ralph Kimball, et al; and Business Intelligence Roadmap  by Larissa Moss and Shaku Atre. In them, the authors speak quite clearly about taking an iterative approach to delivery, the importance of business participation and how prototyping is an important step in the process. Oh, and by the way, these books were published 8-10 years ago!

Now, let’s go back to the two stories. In both cases, the project managers and IT groups were following the "big bang” approach. This approach has been debunked so often I really don’t know who to give credit to. I do remember reading a number of articles and bi-folds on “The 10 Reasons Data Warehouse/BI Projects Fail” and other similar publications from organizations such as TDWI. More times than not, “big bang,” lack of business involvement, lack of data standards and data-driven approaches made everyone’s top 10 list. Using the business requirements to drive candidate source system selection? Not new and certainly not radical. Creating prototypes to gain clarity and expanding the project outward from there (aka spiral)? Not new!

So what is radical and new? From my perspective, nothing about Agile BI is radical. Once you get comfortable with it, you will quickly conclude that it’s mostly common sense. What’s new is applying Agile/Scrum methods to BI projects.

The following diagram provides a context for the Agile BI delivery process that I helped architect for my own company, Information Control Corporation.

SOURCE: The 12 (or so) Myths about Agile Business Intelligence – Myth 1

Recent articles by Jim Gallo

• Applying Agile Methods to Data Warehouse Projects
• The 12 (or so) Myths about Agile Business Intelligence – Myth 4
• The 12 (or so) Myths about Agile Business Intelligence – Myth 3
• The 12 (or so) Myths about Agile Business Intelligence – Myth 2