The Business is Ready to Drive Business Intelligence
The recent economic challenges have taught us that our business leaders need to do more with the resources they already have. One area in which this is strongly apparent is their data warehouse (DW) and business intelligence (BI) systems. Data is now viewed as a critical corporate resource. Turning that data into meaningful information that can support vital decision making has become an absolute mandate throughout the organization.
In Jim Gallo’s article “Business Context for Agile Business Intelligence”, we gain an appreciation for how an agile, iterative approach to building BI systems makes sense. It allows us to avoid the costly time-consuming traps often associated with traditional a SDLC approach, while at the same time allowing the business to be engaged and focused on their priorities (driving revenues, reducing costs, improving customer service, and maintaining financial controls).
Yes, the business is ready to drive our priorities in order to get the information they need to solve their specific challenges. Now what about all of those databases, servers, business rules, ETL jobs, semantic layers, and visualizations that will need to be built? They’re way more than simply “a means to an end” and they will likely consume a good deal of our project time and work effort. We need a way to align our IT deliverables to the business-driven benefits we want from Agile BI.
The Challenge of Business Requirements
Some challenges of business requirements:
- In an Agile BI environment, we look to our business partners to provide requirements that drive our agenda
- These BI requirements, coming from the business, are often stated at a high-level and overly generalized format
- It’s difficult to identify and align the priorities for the various DW and BI components that we’re going to need to build
- It’s easy to lose our business driven mandate and simply focus too much on incorporating new sources into the data warehouse
We need a process where our business requirements gathering and our technical build processes can work together in a loosely coupled way.
Understanding the Value Chains
The agile methods approach allows us to focus on two distinct “value chains”:
- The Business Value Chain: We deconstruct those high-level business requirements into working dimensional business models that become the foundation of our BI semantic layer.
- The Technical Value Chain: The data architecture and data integration components are built that move data from our source systems, into the data warehouse, and ultimately deliver it to our BI applications.
“Scope” in the Technical Value Chain
A primary advantage to adopting agile and iterative DW/BI development principals is that they allow us to adjust and realign work products as the business adjusts and realigns their priorities. This is possible because developers are working from a dynamic backlog of “stories” (Extract-transform-load [ETL] jobs and data components) that establish our Technical Value Chain. In future articles, I’ll focus on building a governance process that will enable us to align the two value chains. For now, we’ll focus on creating those stores necessary to build the deliverables needed in the technical value chain.
Understanding the Components to Building our Technical Value Chain Backlog (1-2-3)
- Understand the underlying architecture needed to deliver data warehouse/business intelligence solutions
Every DW/BI solution has an underlying architecture for how data moves from source systems to their ultimate destination, where it can be consumed. The following is a typical example:
- Source to staging
- Staging to enterprise data warehouse
- Enterprise data warehouse (EDW) to dimensional data marts
- Data mart presented in a semantic layer (available for queries, reports, and dashboards)
All of the potential story types that need to be built to support this architecture can be identified.
- Perform source system assessment (one per source table)
- Develop source to staging ETL job (one per source table)
- Complete DW modeling
- Develop staging to EDW ETL job(s)
- Design data mart views (one per target)
- Develop OLAP cubes
- And so on…
We can estimate them in detail later when we organize our stories into a scrum plan. For now we’ll just identify them and put them into our initial product backlog (sometimes called the “ice box”).
- Understand that scope in the technical value chain will always be driven by the “Number of Objects” that can be built.
Remember that simple scoping project management principle: Resources x Time = Deliverables? The hardest part of that equation has always been identifying the deliverables that need to be built. Starting with a good set of stories (“deliverables”) is essential for getting our agile DW/BI plans in motion. A useful high-level estimate can be done once we know how many “things” need to be built.
- Understand how the data mart is the “bridge” between the business and technical chains
The dimensional business model built in the business value chain becomes the foundation for the business intelligence solution. It provides the semantic layer used for BI queries, reports, and dashboards as well as the underlying “STAR” schema used in our data mart design.
The data mart is also the final destination for data moving through the technical value chain. Based on our delivery architecture that actual data mart may be any combination of tables, views, or cubes designed specifically for business consumption.
We can use the data mart components as targets for the technical value chain delivery architecture.
- Use the business value chain as a vehicle to gather and vet requirements, build and prototype BI business models, and provide the basis for the data mart solution.
- Focus on building those dimensional business models first (in the business value chain) then use that model to drive your technical value chain iterations (releases).
- The agile scrum stories (work components) needed in the technical value chain is based on two things: (1) the target data mart dimensions and facts and (2) your specific architecture for moving data through the data warehouse components.
- This set of agile scrum stories becomes the product backlog of components that need to be built to support the DW/BI release.
SOURCE: Building Agile Scrum Plans from Business Intelligence Requirements