We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

The 12 (or so) Myths about Agile Business Intelligence – Myth 2 There is No Need to Produce Documentation When Using Agile BI Methods

Originally published October 26, 2010

This article is the second in a series that will address the 12 myths about Agile business intelligence (BI). You can find the complete list on my BeyeNETWORK blog. You can read the full article on Myth 1: Agile BI is Radical and New on my ByeNETWORK Agile BI Channel.

I’m not sure why some folks believe that using Agile BI methods implies that there is no need to produce documentation, and believe me, I’ve heard this many times. Perhaps it stems from the 10th point of the Agile Manifesto, “Simplicity – the art of maximizing the amount of work not done – is essential.” However, my reading of the Agile Manifesto certainly doesn’t cause me to draw that conclusion. I do think, however, that we can take a lesson from “maximizing the amount of work not done” that I’ll discuss later in this article.

Let’s get real! Regardless of the delivery method used, whether it’s waterfall, iterative or Agile, we must produce documentation. Documentation exists in many forms – project charters, project plans, requirements and design specifications, testing documents and production support procedures, to name a few. Documentation serves a very important purpose in that it allows us to communicate and commemorate decisions and outcomes (scope, design, operating procedures, etc.) needed to build, enhance and maintain the solution. Without it, we would not be able to support the delivered solution, and it would be extremely difficult to communicate outside the Agile Scrum team without it. Finally, documentation is needed, if for no other reason, than to stay in compliance with laws and mandates such as the Sarbanes-Oxley Act of 2002HIPAA (Health Insurance Portability and Accountability Act of 1996), and Basel II.

I’ve also been asked if Agile BI suggests that we should create “light documentation.” Even if I knew what that meant, the answer is still a resounding “No!” The Agile teams that I’ve been involved with have created some very detailed design and test specifications that would make any CMMI level 5 shop jealous. And what’s really cool, the more detailed the specifications, the lower the defect rate. Don’t misunderstand, I still value “working code” over documentation; I just think we need to strike the right balance of documentation to achieve the highest team velocity.

When architecting the Agile BI approach used by my business intelligence services company, ICC, we made sure to include documentation needs in both our estimating methods (project and story point estimating) and in our tracking charts (project performance metrics). As a matter of practice, we create separate story cards to account for documentation. For example, we might create a Business Value Chain story card titled, “Create User Guide” and a Technical Value Chain story card might be titled, “Create Design Specification for Type 2 Dimension.” In doing this, we can regularly allocate hours and/or story points to documentation tasks.

Whether your objective is to provide business intelligence for sales, business intelligence for marketing or business intelligence for CRM, I do believe there are better ways to create and maintain documentation around certain artifacts such as requirements specifications, dimensional database designs, business glossaries and ETL mapping specifications. We’ve historically created these documents in Microsoft Word, Excel and design tools such as ERwin. We regularly hand over 30 to 100 page documents to the business for review and approval, hoping to make sure that it covers everything they could ever ask for. In other words, we bury their needs.

To make matters worse, we then walk them through data models, asking them to validate that the model is correct. We hand them something like this…

(mouseover image to enlarge)

…and expect them to “see the business” through the model. The problem with the Word documents and the data models is that we are expecting the business to see itself as we in the IT profession do, when in reality the business may have only a faint idea of what they are being asked to review and approve. The reality is that the business sees itself like this:

…and not as a series of words on a page or as an entity-relationship diagram. And what’s even more important to recognize is that the historically verbose and technically bent documents have a life span that’s shorter than a mayfly.

Be of good cheer! A new class of tools is rapidly emerging that allows BI teams to capture, model and maintain business information in ways that are accessible, reusable and, most importantly, understandable. The best one I’ve seen is Balanced Insight’s Consensus that’s been woven into my company’s Agile BI delivery model. I am absolutely convinced that using tools such as Consensus drives down overall delivery costs and enhances a Scrum teams’ ability to maximize the amount of work not done through elimination of redundancy and rework and through the production of outstanding artifacts and documentation.

While I’m on the topic of “maximizing the amount of work not done,” there is something that Agile BI teams (heck, all BI delivery teams), need to take to heart. There is no value in sourcing data for data’s sake – something that seems to be more pervasive than waterfall projects. If you believe, as I do, that 60% to 70% of business intelligence/data warehousing (BI/DW) delivery and the vast majority of project risk is associated with data integration and data quality, why, for Pete’s sake, would anyone in their right mind source more data than is needed to answer the business questions (as defined on the story cards)?!!!! Take another look at the Agile BI process I introduced in debunking Myth 1.


(mouse over image to enlarge)

Note the transition from the Business Value Chain to the Technical Value Chain. Candidate sources are identified from the business terms. Source analysis and data profiling tasks stem from the business stories and can be traced back to them. In other words, if there is no requirement to source specific data sets, then DON’T DO IT as it only adds to project costs and risks. Even if you do not see a need to adopt Agile BI methods, I really encourage you to adopt this particular practice.

To sum it up, Agile BI methods do indeed produce documentation needed for design, test, support and compliance. Documentation gets planned along with each Sprint and the documentation tasks are expressed as individual story cards. While we still can’t avoid creating some documentation in Word, Excel, PowerPoint and Visio, tools are available today that can alleviate some pain associated with producing and maintaining documentation. Just as importantly, these tools can be woven into the Agile BI delivery process to help improve velocity and decrease defects.

Look for my next article on Myth 3 – Agile BI methods do not support sound BI/DW architecture principles. Cheers!
  • Jim GalloJim Gallo
    Jim is  National Director, Vice President, Business Analytics at Information Control Corporation (ICC), a firm focused on reducing the cost of developing BI solutions. Jim and the ICC team have enabled companies to increase the velocity of their development by adopting Agile methods for BI, with a particular focus on turning theory into reality.

    Jim is a recognized expert and has published a number of articles on the practical realities of business intelligence and data warehousing. He is a regular speaker at conferences and industry-related events. He has led a number of large, complex BI projects for Fortune 1000 companies in addition to delivering value to federal and state governments and international clients. For two years in a row, Jim has been named IBM Champion for Information Integration and Federation. He can be reached at jgallo@iccohio.com.

    Editor's Note: More articles and resources are available in Jim's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

Recent articles by Jim Gallo



Want to post a comment? Login or become a member today!

Be the first to comment!