I have yet to meet a business intelligence (BI) team that has too little on its plate. In this article, we will look at some of the drivers creating all this work and offer a set of principles that can be used to become more efficient and effective while still focused on delivering value. The main idea for this methodology evolved while leading a BI team for a leading medical manufacturer. We were frustrated by the amount of time spent maintaining our existing infrastructure and the number of features/data elements being added to the data warehouse that weren’t being utilized. During that time, the company was advocating Lean manufacturing principles and practices in our European divisions, and there appeared to be some parallels to our situation. Our dabbling into Lean was the original root that grew into this concept.

Why Is There So Much Pressure on BI Programs?

When calculating the return on investment (ROI) of BI programs and BI projects, we tend to underestimate resources. BI teams aren’t generally built; they evolve over iterations. We learned a number of years ago that the big-bang approach to data integration simply doesn’t work. However, the alternative is that our programs slowly grow over time. We add more reports, more users, more ETL jobs, more data, more applications, more hardware and more overhead. Iterative development also causes architectural challenges. These include, but are not limited to:

• Non-scalable systems
  
  
• Rigid architectures
  
  
• Duplicate metadata
  
  
• Disparate scheduling
  
  
• Unleveraged metadata
  
  
• Additional and dis-integrated applications
  
  
• Multiple security architectures
  
  
• Plug-and-play stovepipe solutions
  
  
• Non-existent data/systems governance
  

Yet, organizations often aren’t receptive to adding more resources to the BI team because the program was sold on reducing the number of resources and time required for reporting and analysis. The general solution is to push the work out to the functional areas of the organization by recruiting subject-matter experts and data stewards. While this helps with governance, it rarely reduces the amount of pressure on BI departments.

Another challenge is that integration and reporting solutions generally take longer than users are willing to wait. Vendors recognize this and offer point, packaged or pre-modeled solutions that are sold directly to functional areas. These often include the hosted (Saas/ASP) solutions that are marketed as fast, simple and easy to use. These solutions are very tempting because they offer the opportunity to deliver reporting and analysis in significantly less time than BI groups can generally promise. However, they usually only target a specific subject area of data; and over time, the functional areas inevitably require integration with other types of data. This causes more work for BI programs as they become entangled in supporting these solutions.

In many organizations, users require access to both strategic and operational (lower latency) data. Heterogeneous data access and enterprise information integration (EII) solutions are gaining popularity and adding complexity to our BI environments. Also, companies are utilizing hosted solutions for operational needs and purchasing large amounts of external data for use in their organization. In many cases, the companies supplying external data also offer their own reporting solutions and refuse to make their data available to BI programs. This can be a challenge to maintain as users begin to require additional functionality and integration with existing BI applications.

What is Lean BI?

Before we delve into what Lean BI is, it is important to address what Lean BI is not. Many people hear the word “lean” and it conjures up images of featureless tools, limited budgets, reduced development and the elimination of jobs. Dispelling those myths out of the gate is crucial in order to garner support from the organization and the BI team for implementing Lean BI. If team members feel that by becoming lean they are working themselves out of a job, then they will not support your efforts. If your customers feel that they will receive less service or be relegated to using suboptimal tools, then they may not support your efforts as well.

So, what is Lean BI? Lean BI is about generating additional value by accomplishing more with existing resources by eliminating waste. Lean BI is a set of principles and practices that have been influenced by 3 main concepts:

1. Lean Manufacturing
   
   
2. Systems Theory
   
   
3. Agile Project Management
   

Lean Manufacturing is a set of principles and practices that evolved from the Toyota Production System. It has helped Toyota become one of the top car manufacturers in the world, and many other companies worldwide have also benefited from Lean Manufacturing. It focuses on identifying customer value and then delivering more with existing resources by eliminating waste in the organization. Waste is defined as any human activity that absorbs resources but creates no value.¹ By freeing up time usually consumed by wasted effort, organizations can focus on spending more time creating customer value. Not all Lean Manufacturing concepts apply to business intelligence and not all Lean BI concepts apply to manufacturing. While some of the principles are more closely aligned with the shop floor, others are universal to all functional areas.

Waste in BI programs is defined as any activity, task, process, mapping, object, code, report or data that absorbs resources but creates no incremental value to the customer. In my experience, BI programs are affected by both structural forms of waste and local forms of waste. Structural forms of waste exist that are more difficult to change such as compliance activities, processes applied universally to very different activities, vendor impositions and hierarchical control. These create additional work for the BI team that often produces little value to the customer. They include waiting for management signoff, waiting for external QA resources to become available and maintaining little used functional reporting applications. Local forms of waste are activities that we engage in that are within the BI team’s control to change without engaging other areas of the organization. Examples of local waste include maintaining the same business rules and metadata elements in more than one application, ETL process steps that are no longer required and avoidable mistakes that require rework.

System theory focuses on the theory, methods and philosophy needed to analyze the behavior of systems in management and other fields.² It recognizes that systems, such as BI programs, are complex and that we must consider this fact when developing our architecture and making decisions. A system consists of interdependent and interacting parts joined by a purpose.³ The ability of a system to achieve its purpose depends on how well the parts work together, not just how they work individually. Systems theory also recognizes that the decisions we make and actions we take affect other parts of the system that we don’t necessarily consider. Decisions made in other parts of the organization often affect BI teams, such as deciding to contract with an outside vendor to host an application that doesn’t allow access to their data. BI programs grow through iterations so we need to consider how our architecture will change over time so we can augment versus replace.

Agile Project Management evolved from the Agile Manifesto for Software Development. Many people have written about Agile, including myself, and its application to BI programs and projects (see Who Doesn’t Want to be Agile? ). Agile was originally designed for software development, but many of its concepts can be applied to BI projects. It is important for BI teams to be agile because businesses are not static, and we must be able to effectively deliver value in a changing environment. Effectively delivering value means reducing the time between request and rollout and only delivering what is required. The challenge with the traditional waterfall approach is that between the time of requirements analysis and delivery, many of the requirements are no longer required because the business has changed. This is a great example of waste in BI programs.

Summary

Delivering more value with existing resources is at the core of Lean BI. As the economy tightens, BI programs will need to adopt new principles and practices if they are to flourish. In future articles, I will reveal the 6 principles of Lean BI and discuss the real-world practices, from the trenches, that can be employed to reduce waste in any BI program or project.

End Notes:

1. Womack and Jones, Lean Thinking
   
   
2. Forrester, “Systems Thinking and the Lessons of the Last 35 Years
   
   
3. Poppendieck, Lean Software Development
   
   

Recent articles by Steve Dine

• Enhancing Your Enterprise Reporting Arsenal with Multidimensional Expressions (MDX)
• Data Warehouse Appliances: Lessons from the Trenches
• One Source of the Truth: Common Business Intelligence Terms Defined
• Who Doesn't Want to be Agile?