I have yet to meet a business intelligence (BI) team that has too little on its plate. In this article, we will look at some of the drivers creating all this work and offer a set of principles that
can be used to become more efficient and effective while still focused on delivering value. The main idea for this methodology evolved while leading a BI team for a leading medical manufacturer. We
were frustrated by the amount of time spent maintaining our existing infrastructure and the number of features/data elements being added to the data warehouse that weren’t being utilized.
During that time, the company was advocating Lean manufacturing principles and practices in our European divisions, and there appeared to be some parallels to our situation. Our dabbling into Lean
was the original root that grew into this concept.
Why Is There So Much Pressure on BI Programs?
When calculating the return on investment (ROI) of BI programs and BI projects, we tend to underestimate resources. BI teams aren’t generally built; they evolve over iterations. We learned
a number of years ago that the big-bang approach to data integration simply doesn’t work. However, the alternative is that our programs slowly grow over time. We add more reports, more users,
jobs, more data, more applications, more hardware and more overhead. Iterative development also causes architectural challenges. These include, but are not limited to:
- Non-scalable systems
- Rigid architectures
- Duplicate metadata
- Disparate scheduling
- Unleveraged metadata
- Additional and dis-integrated applications
- Multiple security architectures
- Plug-and-play stovepipe solutions
- Non-existent data/systems governance
Yet, organizations often aren’t receptive to adding more resources to the BI team because the program was sold on reducing the number of resources and time required for reporting and
analysis. The general solution is to push the work out to the functional areas of the organization by recruiting subject-matter experts and data stewards. While this helps with governance, it rarely
reduces the amount of pressure on BI departments.
Another challenge is that integration and reporting solutions generally take longer than users are willing to wait. Vendors recognize this and offer point, packaged or pre-modeled solutions that are
sold directly to functional areas. These often include the hosted (Saas/ASP) solutions that are marketed as fast, simple and easy to use. These solutions are very tempting because they offer the
opportunity to deliver reporting and analysis in significantly less time than BI groups can generally promise. However, they usually only target a specific subject area of data; and over time, the
functional areas inevitably require integration with other types of data. This causes more work for BI programs as they become entangled in supporting these solutions.
In many organizations, users require access to both strategic and operational (lower latency) data. Heterogeneous data access and enterprise information integration (EII) solutions are gaining
popularity and adding complexity to our BI environments. Also, companies are utilizing hosted solutions for operational needs and purchasing large amounts of external data for use in their
organization. In many cases, the companies supplying external data also offer their own reporting solutions and refuse to make their data available to BI programs. This can be a challenge to maintain
as users begin to require additional functionality and integration with existing BI applications.
What is Lean BI?
Before we delve into what Lean BI is, it is important to address what Lean BI is not. Many people hear the word “lean” and it conjures up images of featureless tools, limited
budgets, reduced development and the elimination of jobs. Dispelling those myths out of the gate is crucial in order to garner support from the organization and the BI team for implementing Lean BI.
If team members feel that by becoming lean they are working themselves out of a job, then they will not support your efforts. If your customers feel that they will receive less service or be
relegated to using suboptimal tools, then they may not support your efforts as well.
So, what is Lean BI? Lean BI is about generating additional value by accomplishing more with existing resources by eliminating waste. Lean BI is a set of principles and practices that have been
influenced by 3 main concepts:
- Lean Manufacturing
- Systems Theory
- Agile Project Management
Lean Manufacturing is a set of principles and practices that evolved from the Toyota Production System. It has helped Toyota become one of the top car manufacturers in the world, and many other
companies worldwide have also benefited from Lean Manufacturing. It focuses on identifying customer value and then delivering more with existing resources by eliminating waste in the organization.
Waste is defined as any human activity that absorbs resources but creates no value.1
By freeing up time usually consumed by wasted effort, organizations can focus on spending more time
creating customer value. Not all Lean Manufacturing concepts apply to business intelligence and not all Lean BI concepts apply to manufacturing. While some of the principles are more closely aligned
with the shop floor, others are universal to all functional areas.
Waste in BI programs is defined as any activity, task, process, mapping, object, code, report or data that absorbs resources but creates no incremental value to the customer. In my experience, BI
programs are affected by both structural forms of waste and local forms of waste. Structural forms of waste exist that are more difficult to change such as compliance activities, processes applied
universally to very different activities, vendor impositions and hierarchical control. These create additional work for the BI team that often produces little value to the customer. They include
waiting for management signoff, waiting for external QA resources to become available and maintaining little used functional reporting applications. Local forms of waste are activities that we engage
in that are within the BI team’s control to change without engaging other areas of the organization. Examples of local waste include maintaining the same business rules and metadata elements in
more than one application, ETL process steps that are no longer required and avoidable mistakes that require rework.
System theory focuses on the theory, methods and philosophy needed to analyze the behavior of systems in management and other fields.2
It recognizes that systems, such as BI programs, are
complex and that we must consider this fact when developing our architecture and making decisions. A system consists of interdependent and interacting parts joined by a purpose.3
ability of a system to achieve its purpose depends on how well the parts work together, not just how they work individually. Systems theory also recognizes that the decisions we make and actions we
take affect other parts of the system that we don’t necessarily consider. Decisions made in other parts of the organization often affect BI teams, such as deciding to contract with an outside
vendor to host an application that doesn’t allow access to their data. BI programs grow through iterations so we need to consider how our architecture will change over time so we can
augment versus replace.
Agile Project Management evolved from the Agile Manifesto for Software Development. Many people have written about Agile, including myself, and its application to BI programs and projects (see
Who Doesn’t Want to be Agile?
was originally designed for software development, but many of its concepts can be applied to BI projects. It is important for BI teams to be agile because businesses are not static, and we must be
able to effectively deliver value in a changing environment. Effectively delivering value means reducing the time between request and rollout and only delivering what is required. The challenge with
the traditional waterfall approach is that between the time of requirements analysis and delivery, many of the requirements are no longer required because the business has changed. This is a great
example of waste in BI programs.
Delivering more value with existing resources is at the core of Lean BI. As the economy tightens, BI programs will need to adopt new principles and practices if they are to flourish. In future
articles, I will reveal the 6 principles of Lean BI and discuss the real-world practices, from the trenches, that can be employed to reduce waste in any BI program or project.
- Womack and Jones, Lean Thinking
- Forrester, “Systems Thinking and the Lessons of the Last 35 Years
- Poppendieck, Lean Software Development
Recent articles by Steve Dine