The first part of this blog examines the pros and cons of centralized, decentralized, and federated BI architectures using Intuit as a case study. The second part looks at organizational models and how they influence BI architectures using Harley-Davidson and Dell as case studies. The third part shows different ways that corporate and divisional groups can divide responsibility for BI architectural components in the context of various organizational models.
PART I
The Federated Option
"Business needs trump architectural purity every time." Douglas Hackney
For years, business intelligence (BI) experts have diagrammed data warehousing architectures in which a single set of enterprise data is extracted and transformed from various operational systems and loaded into a staging area or data warehouse that feeds downstream data marts. This classic architecture is designed to deliver a single version of high-quality enterprise data while tailoring information views to individual departments and workgroups. It also ensures an efficient and scalable method for managing the flow of operational data to support reporting and analysis applications.
That's the theory, anyway. Too bad reality doesn't work that way.
Federated Reality. Most organizations have a hodgepodge of insight systems and non-aligned reporting and analysis applications that make achieving the veritable single version of truth a quest that only Don Quixote could appreciate. Let's face it, our BI architectures are a mess. But this is largely because our organizations are in a constant state of flux. If an ambitious, tireless, and lucky BI team achieved architectural nirvana, their edifice wouldn't last more than a few months. Mergers, acquisitions, reorganizations, new regulations, new competitors, and new executives with new visions and strategies all conspire to undermine a BI manager's best laid architectural blueprints.
Thus, the BI reality in which most of us live is federated. We have lots of overlapping and competing sources of information and insight, most of which have valid reasons for existence. So rather than trying to roll a boulder up a hill, it's best to accept a federated reality and learn how to manage it. And in the process, you'll undoubtedly discover that federation--in most of incarnations--provides the best way to balance the need for enterprise standards and local control and deliver both efficient and effective BI solutions.
Understanding the Options - Evolution of Intuit's BI Architecture
Federation takes the middle ground between centralized and decentralized BI approaches. Ideally, it provides the best of both worlds, while minimizing the downsides. Table 1 describes the pros and cons of centralized and decentralized approaches to BI. Not surprisingly, the strengths of one are the weaknesses of the other, and vice versa.
Table 1. Centralized versus Decentralized BI
Founded in 1983, Intuit is a multi-billion dollar software maker of popular financial applications, including Quicken, TurboTax, and QuickBooks. Intuit's BI evolution helps to illuminate the major characteristics of the two approaches and explains how they ended up with a federated BI architecture. (See figure 1.)
Decentralized Approach. In the late 1990s, after a series of acquisitions, Intuit had a decentralized BI environment. Essentially, every division--most which had been independent companies--handled its own BI requirements without corporate involvement. Divisions with BI experts built their own applications to meet local needs. Deployments were generally quick, affordable, and highly tailored to end user requirements. As true with any decentralized environment, the level of BI expertise in each division varied significantly. Some divisions had advanced BI implementations while others had virtually nothing. In addition, there was considerable redundancy in BI software, staff, systems , and applications across the business units, and no two systems defined key metrics and dimensions in the same way.
Centralized Approach. To obtain promised financial synergies from its acquisitions and create badly needed enterprise views of information, Intuit in 2000 implemented a classic, top-down, centralized data warehousing architecture. Here, a newly formed corporate BI team built and managed an enterprise data warehouse (EDW) to house all corporate and divisional data. Divisions rewrote their reports to run against the unified metrics and dimensions in the data warehouse. The goal was to reduce costs through economies of scale, standardize data definitions, and deliver enterprise views of data.
Although Intuit realized these gains, the corporate BI team quickly became overwhelmed with project requests, much to the chagrin of the divisions, which began to grumble about the backlog. When the divisions began to look elsewhere to meet their information requirements, the corporate BI team recognized it needed to reinvent itself and its BI architecture.
Figure 1. Intuit's BI Architectural Evolution
Courtesy Intuit
Federated Approach. To forestall a grassroots resurrection, Intuit adopted a federated architecture that blends the best of top-down and bottom-up approaches. To uncork its project backlog, the corporate BI team decided to open up its data warehousing environment to the divisions. It gave each division a dedicated partition in the EDW to develop their own data marts and reports. It also taught the divisions how to use the corporate ETL tool and allowed them blend local data with corporate data inside their EDW partitions. For divisions that had limited or no BI expertise, the corporate BI team continued to build custom data marts and reports as before. (See figure 2.)
Figure 2. Intuit's Federated Architecture
Benefits. The federated approach alleviated Intuit's project backlog and gave divisions more control over their BI destiny. They could now develop custom tailored applications and reports without having to wait in line. For its part, the corporate BI team achieved economies of scale by maintaining a single platform to run all BI activity and maintained tight control over enterprise data elements. Today, the corporate BI team no longer tries to be the only source of corporate data. Rather, it focuses on capturing and managing data that shared between two or more divisions and develop BI applications when asked.
Table 2. Federated BI
Challenges. The main challenge with federation is keeping everyone who is involved in the BI program, either formally or informally, on the same page. This requires a lot of communication and consensus building, or "social architecture" as Intuit calls it. The corporate BI team needs to relinquish some of its control back to the business units--which is a scary thing for a corporate group to do--while business units need to adhere to basic operating standards when using the enterprise platform. Ultimately, to succeed, federation requires a robust BI Center of Excellence with a well-defined charter, a clear roadmap, strong business-led governance, an active education program, and frequent inhouse meetings and conferences. (See Table 2.)
Ultimately, managing a federated environment is like walking a tightrope. You have to balance yourself to keep from swinging too far to the left or right and falling off the rope. To succeed, you have to make continuous adjustments, ensuring that everyone on the extended BI team is moving in the same direction. Without sufficient communication and encouragement, a federated approach courts catastrophe.
 
PART II
Organizational Models That Impact BI
"Architecture must conform with corporate culture and evolve with the prevailing winds."
In Part I of this series, we discussed the strengths and weaknesses of different architectural options, including federation. And we used Intuit as an example of an organization that evolved through decentralized and centralized architectural models until it settled on federation as a means to marry the best of both worlds.
It's important to recognize that the real problem that federation addresses is not architectural; it's organizational. There is natural tension between local and central organizing forces within a company. Every organization needs to give local groups some degree of autonomy to serve customers effectively. Yet it also needs a central, guiding force to achieve economies of scale (efficiency), present a unified face to the customer (effectiveness), and deliver enterprise views of corporate performance. BI, as an internal service, needs to reflect a company's organizational structure. But knowing where to draw the line architecturally is challenging, especially as a company's business and organizational strategy evolves.
Organizational Models
At a high level, there are three basic organizational models that define the relationship between corporate and divisional groups: Conglomerate, Cooperative, and Centralized. Although, this article treats each as a unique entity, the models represent points along a spectrum of potential organizational structures. The key to each model's identity and defining characteristics, however, stems from a company's product strategy. (See table 3.)
Conglomerate Model. A conglomerate, for example, has a highly diversified product portfolio that spans multiple industries. Here, each division sells radically different products to very different customers. For example, General Electric, is a conglomerate. It sells everything from financial services to household appliances to wind turbines. A conglomerate typically gives business unit executives complete autonomy to run their operations as they see fit using their own money and people. There is little shared data and each business unit has its own BI team and technology and exchanges little, if any, data with other business units.
Table 3. Organizational Models
Centralized Model. The centralized model falls at the other end of the organizational spectrum. Here, an organization sells one type of product to anyone, everywhere. As a result, corporate plays a strong hand in defining product and marketing strategies while business units primarily execute sales in their regions. In a centralized model, IT and BI are shared services that run centrally at corporate. There is one BI team and enterprise data warehouse that supports all business units and users.
Dell, for example, sells personal computers and servers on a global basis. Although its regional business units had significant autonomy to meet customer needs during the company's rapid expansion, Dell now has retrenched. Without dampening its entrepreneurial zeal, Dell has strengthened its centralized shared services, including BI, to streamline redundancies, reduce overhead costs, improve cross-selling opportunities, and deliver more timely, accurate, and comprehensive views of enterprise performance to top executives.
Cooperative Model. The Cooperative model--which is the most pervasive of the three--falls somewhere between the Conglomerate and Centralized models on the organizational spectrum. Here, business units sell similar, but distinct, products to a broad set of overlapping customers. Consequently, there are significant cross-selling opportunities, which require business units to share data and work cooperatively for the greater good of the organization. For example, Intuit, with its various financial software products, can increase revenues by cross-selling its various consumer and business products across divisions. Most financial services firms, which offer banking, insurance, and brokerage products, also employ a Cooperative model.
Although most divisions in a Cooperative model maintain BI teams, corporate headquarters also fields a sizable corporate BI group. Given the distribution of BI teams and technology across divisions, the Cooperative model by default requires a federated approach to BI. The challenge here is figuring out how to divide the BI stack between corporate and divisional units. There are many places to draw the line. Often, the division of responsibility is not clear cut and changes over time as a company evolves its business strategy to meet new challenges.
Harley-Davidson. For example, Harley-Davidson Motor Co., an iconic, U.S.-based manufacturer of motorcycles, decided to broaden its business strategy after the recent recession to focus more on global markets, e-commerce, and custom-made products. Consequently, the BI team created a new strategy to meet these emerging business requirements that emphasizes the need for greater agility and flexibility.
Currently, Harley-Davidson has well-defined, centralized processes for sourcing, validating and delivering data. Any application or project that requires data must be approved by the Integration Competency Center (ICC), which generally also performs the data integration work. The ICC's job is to ensure clean, consistent, integrated data throughout the company.
Currently, the ICC handles all business requests the same way, regardless of the nature or urgency of the request. In the future, to go as fast as the business wants, the ICC will break free from its "one size fits all" methodology and take a more customized approach to business requests. "We need to make sure our processes don't get in the way of the business," said Jim Keene, systems manager of global information services.
Dell. While Harley-Davidson is loosening up, Dell is tightening up. After experiencing rapid growth during the 1990s and 2000s through an entrepreneurial business model that gave regional business units considerable autonomy, the company now wants to achieve both fast growth and economies of scale by strengthening its central services, including BI. In the mid-2000s, Dell had hundreds of departmental systems, thousands of BI developers, and tens of thousands of reports that made it difficult to gain a global view of customers and sales.
As a result, Dell kicked off an Enterprise BI 2.0 initiative chartered to consolidate and integrate independent data marts and reports into a centralized data warehouse and data management framework. It also created a BI Competency Center that oversees centralized BI reporting teams that handle report design and access, metric definitions, and governance processes for different business units.. By eliminating redundancies, the EBI program has delivered a 961% return on investment, improved data quality and consistency, and enhanced self-service BI capabilities.
While Harley-Davidson and Dell represent two ends of the spectrum--a centralized BI team that is rethinking its centralized processes and a decentralized BI environment that is consolidating BI activity and staff--most organizations are somewhere in the middle. Part III of this article will map our three organizational models to a BI stack and show the various ways that a BI team can allocate BI architectural elements between corporate and divisional entities.
 
Part III
The Dividing Line
"If it can't be my design, tell me where do we draw the line." Poets of the Fall
In Part II of this article, we discussed the impact of organizational models on BI development. In particular, we described Conglomerate, Cooperative, and Centralized models and showed how two companies (Harley-Davidson and Dell) migrated across this spectrum and its impact on their BI architectures and organizations.
As any veteran BI professional knows, BI architectures come in all shapes and sizes. There is no one way to build a data warehouse and BI environment. Many organizations try various approaches until they find one that works, and then they evolve that architecture to meet new business demands. Along the way, each BI team needs to allocate responsibility for various parts of a BI architecture between corporate and business unit teams. Finding the right place to draw the proverbial architectural line is challenging.
The BI Stack. Figure 3 shows a typical BI stack with master data flowing into a data warehouse along with source data via an ETL tool. Data architects create business rules that are manifested in a logical model for departmental marts and business objects within a BI tool. BI developers who write code and assemblers who stitch together predefined information objects, create reports and dashboards for business users. Typically, most databases and servers that power operational and analytical systems run in a corporate data center.
Using this high-level architectural model, we can study the impact of the three organizational models described in Part II of this article on BI architectures.
Figure 3. Typical BI Stack
Conglomerate - Shared Data Center. In a Conglomerate model, business units have almost complete autonomy to design and manage their own operations. Consequently, business units also typically own the entire BI stack, including the data sources, which are operational systems unique to the business unit. Business units populate their own data warehouses and marts using their own ETL tools and business rules. They purchase their own BI tools, hire their own BI developers, and develop their own reports. The only thing that corporate manages is a data center which houses business unit machines and delivers economies of scale in data processing. (See figure 4.)
Figure 4. Conglomerate BI
Cooperative BI - Virtual Enterprise. In a Cooperative model where business units sell similar, but distinct, products, business units must work synergistically to optimize sales across an overlapping customer base. Here, there is a range of potential BI architectures based on an organization's starting point. In the Virtual Enterprise model, an organization starts to move from a Conglomerate business model to a more Centralized model to develop an integrated view of customers for cross-selling and upselling purposes and maintain a single face to customers who purchase products from multiple business units.
In a Virtual Enterprise, business units still control their own operational systems, data warehouses, data marts, and BI tools and employ their own BI staff. Corporate hasn't yet delivered enterprise ERP applications but is thinking about it. Its first step towards centralization is to identify and match mutual customers shared by its business units. To do this, the nascent corporate BI team develops a master data management (MDM) system, which generates a standard record or ID for each customer that business units can use in sales and service applications. (See figure 5.)
Figure 5. Cooperative BI - Virtual Enterprise
The corporate group also creates a fledgling enterprise data warehouse to deliver an enterprise view of customers, products, and processes common across all business units. This enterprise data warehouse is really a data mart of distributed data warehouses. That is, it sources data from the business unit data warehouses, not directly from operational systems. This can be a persistent data store populated with an ETL or virtual views populated on the fly using data virtualization software. A persistent store is ideal for non-volatile data (i.e., dimensions that don't change much) or when enterprise views require complex aggregations, transformations, or multi-table joins, or large volumes of data. A virtual data store is ideal for delivering enterprise views quickly at low cost and building prototypes and short-lived applications.
Cooperative BI - Shared BI Platform. The next step along the spectrum is a Shared BI Platform. Here, corporate expands appetite for data processing. It replaces business unit operational applications with enterprise resource planning (ERP) applications (e.g., finance, human resources, sales, service, marketing, manufacturing, etc.) to create a more uniform operating environment. It also fleshes out its BI environment, creating a bonafide enterprise data warehouse which pulls data directly from various source systems, including the new ERP applications, instead of departmental data warehouses. This reduces redundant extracts and ensures greater information consistency. (See figure 6.)
Figure 6. Cooperative BI - Shared Platform
Meanwhile, business units still generate localized data and require custom views of information. While they may still have budget and license to run their own BI environments, they increasingly recognize that they can save time and money by leveraging the corporate BI platform. To meet them halfway, the corporate BI team forms a BI Center of Excellence and teaches the business units how to use the corporate ETL tool to create virtual data marts inside the enterprise data warehouse. The business units can upload local data to these virtual marts, giving them both enterprise and local views of data without having to design, build, and maintain their own data management systems.
In addition, the corporate BI team builds a universal semantic layer of shared data objects (i.e., a semantic layer) that it makes available within the corporate BI tool. Although business units may still use their own BI tools, they increasingly recognize the value of building new reports with the corporate BI tools because they provide access to the standard business objects and definitions, which they are required or highly encouraged to use. The business units still hire and manage their own BI developers and assemblers, but they now have dotted line responsibility to the corporate BI team and are part of the BI Center of Excellence. This is the architectural approach used by Intuit (See Part I.)
Centralized BI - Shared Service. Some organizations, like Dell, don't stop with a Shared Platform model; they continue to centralize BI operations to improve information integrity and consistency and squeeze all redundancies and costs out of the BI pipeline. Here, the corporate BI team manages the entire BI stack and creates tailored reports for each business unit based on requirements. For example, Dell's EBI 2.0 program (see Part II) reduced the number of report developers in half and reassigned them to centralized reporting teams under the direction of a BI Competency Center where they develop custom reports for specific business units. The challenge here, as Harley-Davidson discovered, is to keep the corporate BI bureaucracy from getting too large and lumbering and ensure it remains responsive to business unit requirements. This is a tall task, especially in a fast-moving company whose business model, products, and customers change rapidly.
Summary
Federation is the most pervasive BI architectural model, largely because most organizations cycle between centralized and decentralized organizational models. A BI architecture, by default, needs to mirror organizational structures to work effectively. Contrary to popular opinion, a BI architecture is a dynamic environment, not a blueprint written in stone. BI managers must define an architecture based on prevailing corporate strategies and then be ready to deviate from the plan when the business changes due to an unanticipated circumstance, such as a merger, acquisition, or new CEO.
Federation also does the best job of balancing the twin needs for enterprise standards and local control. It provides enough uniform data and systems to keep the BI environment from splintering into a thousand pieces, preserving an enterprise view critical to top executives. But it also gives business units enough autonomy to deploy applications they need without delay or IT intervention. Along the way, it minimizes BI overhead and redundancy, saving costs through economies of scale.
Twitter 
LinkedIn 
YouTube