This article is the third in a series that will address the 12 myths about Agile BI listed on my BeyeNETWORK blog. Myth 3: Agile BI Methods Do Not Support Sounds BI/DW Architecture Principles
Full articles on the first 2 myths can be found on my Agile BI Channel:
Quite honestly, this myth is the one I struggle with the most because I have a hard time understanding where the individuals stating the objections about Agile BI and architecture are coming from. And they mostly come from architects! (In the spirit of full disclosure, I freely admit that I am a recovering architect and I’m happy to report that I’m on step 4 of my program). From what I have gathered, the most pervasive objection goes something like this: “If you deliver parts of the BI solution in small increments, there’s no way to assure that the pieces will fit together at the end.” Or, if you prefer, “the combined results of Sprints might not end in a unified BI/DW architecture.” Another way I’ve heard the concern (assumption?) expressed is based on the corollary, “Sprints start with the product backlog. Ergo, Agile BI doesn’t consider architectural constructs.”
I really do try to understand these objections, but I find the fact that the objections even exist in the first place to be somewhat confusing. In my feeble mind, it’s like saying that the power grid doesn’t have to exist in order for you to plug in a lamp.
This must exist in order to do this.
(mouseover image to enlarge)
Conversely, it’s like saying that homeowners need to know how electricity works and what’s behind the grid in order to install and use a lamp. In reality, they simply need to decide what kind of lamp they want, where they want to place the lamp, first making sure there’s an outlet nearby, and that the outlet will accommodate the lamp’s plug.
Let’s start at the very beginning (it’s a very good place to start) with a definition. Architecture is the design or structure of something
. When architecting the power grid, the architects need to know the approximate number of buildings
the grid must to support, the expected growth rate, the relative geographical distribution of consumers and that, for the most part, 110v power will be required. However, they don’t need to know the specific location of the buildings, the number of floors each will have, nor do they need to know the exact location of individual wall outlets.
A business intelligence/data warehouse (BI/DW) architecture is equivalent to the blueprint of the power grid. The BI architect needs to determine equivalent things such as the expected total number of users and their location, the number of subjects that might ultimately be addressed (e.g., sales, finance, purchasing, HR, etc.), the potential sources of information, anticipated data volumes, data refresh expectations, etc. These macro “requirements” help IT organizations create the structure, or shell, of the overall solution (the “architecture”) without knowing the exact number of reports, how reports will be laid out, which specific data elements will be needed and how each will be acquired and integrated. Nor do the architects need to know the priority order for content delivery.
These initial analyses and inputs result in macro design decisions such as the types of databases needed (warehouses, marts, operational data stores, etc.), data acquisition and integration methods required (batch processes, real-time updates, change data capture, information busses, etc.), and the potential methods of delivery (reports, scorecards, cubes, etc.). Once the architecture has been determined, tools and technologies that will support the given architecture are selected. It’s through the definition of these key components and selection of tools that the guiding design and construction principles are created and content delivery can begin.
Now, repeat after me, “Agile BI is a delivery methodology.” It is used to define the specific components of the solution, refine them, and ultimately deliver value to the business in the form of metrics, KPIs and analytical solutions that must fit within the architectural blueprint. In other words, in order for the Agile BI team to begin
a project, the architecture on which their solutions will be built and the tools they have at their disposal must exist. It’s during the iterative delivery process that the business and the BI team get to put their mark on the solution, much the same way city planners and home builders do when they begin to define their wants and needs in the context of the power grid. They trust grid has sufficient capacity and is in good working order as they begin to tap into it. And, to continue with my (admittedly trite) analogy, the homeowners get to define more specific things (the refinement process) such as room size, carpeting, cabinetry and lighting fixtures, trusting that the wiring and plumbing have been properly installed.
The same types of processes apply to Agile BI delivery in the context of the value chains.
(mouseover image to enlarge)
Agile BI is really about the decomposition of the business needs using analysis and design techniques (top-down) and the synthesis of the details into architectural components (bottom-up) that deliver value back to the business. The BI solution is the sum of the components designed, built and tested within the value chains. Because Agile BI should not and cannot ignore architectural constructs and standards, a good Agile BI really breaks the Technical Value Chain into two separate, but codependent pieces – the Data Integration and Architecture Value Chains. Each layer of the Agile BI model produces the same components as other delivery methods do within each of the three value chains. It’s just done differently.
(mouseover image to enlarge)
I can already hear the architects out there saying “Aha, but the Agile BI delivery process cannot continue ad infinitum because the architecture and infrastructure must continue to morph and grow.” To which I say, “Wait, there’s more!” Let’s take another look at the overlay of my company’s Agile BI process onto the Moss and Atre delivery framework, which we’ve dubbed as the Information Factory approach.
Information Factory – Agile BI Framework within the Entire SDLC
(mouseover image to enlarge)
What we’ve taught our business intelligence
consultants is that the product backlog gets created early in the delivery lifecycle and is given full consideration in steps 1-3 of the Justification and Initiate phases. More importantly, Step 2, Enterprise Infrastructure Evaluation, has been defined specifically to deal with the new realities of the upcoming project and its associated Sprints. It Is here that the Agile BI team re-synchs with the architects and the infrastructure teams to make sure both can support the latest set of business needs. It’s in step 2 that Moss and Atre specify the following activities:
- Assess existing platform
- Evaluate and select new products
- Write technical infrastructure assessment report
- Expand current platform1
If the architecture and infrastructure are okay, we move on to Project Planning and through the Proceed Gate. If not, more time is spent in step 2 to modify the architecture, select new tools or to plan for more capacity, which, in turn, creates additional tasks (and costs) in the project plan. In other words, there’s a clear recognition that the Agile BI team’s solution is dependent on the architecture and its ability to support new requirements. It’s also why these tasks have been serialized and are predecessors to the iterative tasks performed within the Business and Technical Value Chains.
Let’s Not Forget About Agility
If you haven’t already done so, I’d encourage you to read my article “A Business Context for Agile Business Intelligence.” In it, I describe how Agile BI methods allow IT organizations to quickly provide insight into the dynamic and ever-changing business landscape. The need to adapt quickly is why Agile BI methods work so well. Each new project, each new iteration and each new Sprint allow the business to “reshuffle the deck” with regard to priorities. This means that we must be able to deliver value quickly through iterative delivery methods, which, in turn, implies that the BI/DW architecture needs to be flexible enough to accommodate changes in direction. Most importantly, this means that we need to build out the solution one room at a time, one wall at a time, one outlet at a time.
It’s for these reasons that rigid architectural constructs don’t work very well. For example, from a purely architectural perspective, it seems to make sense to create the entire enterprise data model (EDM) up front. The problem with this is that while the EDM is being created, business needs are changing and the resultant model may not look anything like what the business needs when the rest of the project moves forward. If we agree to basic architecture concepts such as the inclusion of a normalized enterprise data warehouse (EDW) surrounded by dependent data marts and conformed dimensions, we can certainly begin design and construction one subject at a time, one table at a time or one column at a time, for the architecture calls for an integrated whole that we simply build and adjust over time.
To sum it all up, Agile BI is a delivery process that clearly supports sound BI/DW architectural principles, whether you’re delivering business intelligence for insurance, business intelligence for utilities or business intelligence for finance. In order to implement the first project, the architecture must exist, the business intelligence tools selected and the infrastructure created, for without them delivery is not possible. The Agile BI process actually begins at the Justification phase where the architecture, tools and infrastructure are reviewed in the context of the requirements contained in the product backlog.
The iterative components of Agile BI (design, develop, test) begin once the project has passed the Proceed gate, and this is where decomposition and synthesis occur within the value chain processes. And it is through these processes that the more familiar deliverables such as new warehouse tables, data marts, reports, performance management dashboards and the like are created within the overall architecture and that are instantiated through the infrastructure and tools. These components are the equivalent of subdivisions, houses, floors and wall outlets within the power grid analogy, each being constructed and refined within the context of the existing architecture as well as the specified standards.
My next article will address Myth 4 – Agile BI displaces the software development lifecycle (SDLC) in its entirety. See you next time as we head down “The Path to Agility.”
- Business Intelligence Roadmap, by Larissa Moss and Shaku Atre, pp. 61-62
SOURCE: The 12 (or so) Myths about Agile Business Intelligence – Myth 3
Recent articles by Jim Gallo