Top down versus bottom up.

Every ten years, the BI market shifts from reporting to analysis and back again. These are two very different environments serving two very different types of users with different tools and architectures. Unfortunately, most BI professionals and their organizations have yet to grasp this crucial distinction between what I call "top down" and "bottom up" BI. And even fewer understand how to reconcile these two worlds. (See "A BI Marketer's Guide")

In the top-down world, reports provide casual users with answers to predefined questions and are built by IT professionals using a data warehouse. In the bottom-up world, analysis is conducted by power users who use low-cost tools to extract, integrate, and analyze data to answer unanticipated questions. Although the two worlds differ wildly, they are intrinsically connected: analysis begets reports and reports beget analysis.

Cyclical Evolution.

In the 1980s, BI consisted mainly of reports--static, paper-based, and generic--that the IT department built, printed and delivered to all employees. To provide insight into data held hostage by static reports, BI vendors in the 1990s decided to empower business analysts with desktop query and analysis tools, including ad hoc query and OLAP tools and spreadsheets. Then in 2000, after seeing most of their software gather dust on customer shelves and finally recognizing that business analysts only constitute 10-15% of an employee base, BI vendors began focusing again on reports for casual users. But this time, they made the reports visual and interactive, calling them dashboards and scorecards.

Like clockwork, in 2010 the BI market shifted its focus again, moving from reporting back to analysis. We are now in the decade of the "data scientist", who is kind of a conglomeration of a business analyst, statistician, and data savvy Java programmer. BI vendors now tout their discovery tools and analytical platforms that empower business analysts to explore vast volumes of multi-structured data so they can find valuable insights worth millions to their organizations. If history remains constant, the market for big data and analytics will hold strong through 2020.

The Future of BI

So what comes next? It should be clear by now that by 2020 the BI market will shift its emphasis back to reporting. BI vendors will recognize that most employees are not data scientists and don't aspire to be one of them. BI vendors will rediscover the value of top-down reporting and dashboarding for casual users, but with a twist. Reports and dashboards in 2020 will be embedded into other applications rather than separate applications that users must check to obtain information and insights. In 2020, BI becomes largely invisible, embedded into core run-the-business applications, such as sales, marketing, and finance, which access "BI services" through a Web or cloud interface.

But these embedded reports and dashboards will be quite different from those of the 1980s and even the 2000s. They will be much more interactive, analytical, and open ended. Rather than offering canned views of summary data with constricted drill paths, embedded reports in 2020 and beyond will provide the complete range of BI functionality--from reports and OLAP to discovery and mining to dashboarding and authoring--exposed on demand as users need them or are ready and willing to use them.

The BI OEM Market.

Currently, there is a lot of embedded BI if you know where to look. Most independent software vendors (ISVs) embed third party BI tools so customers can view reports and analyze application data. In the 1990s, most ISVs embedded Crystal Reports. Today, ISVs are launching an avalanche of cloud-based applications and thus seek cloud-friendly BI tools to embed in their software. Vendors such as Jaspersoft, LogiXML, and Pentaho currently dominate the BI OEM market, but other fast-growing BI vendors, such as Tableau, Birst, and QlikView, are investing resources to compete in this expanding market.

But the future of embedded BI extends beyond the ISV and OEM market. By 2020, the majority of organizations will embed BI in their own homegrown mission-critical applications. These embedded BI applications will support real-time insight, automated decision making, and action-oriented workflows. These applications of the future will fuse operations and analytics into a seamless, highly adaptive environment that optimizes business processes and delivers deep business value.

I recently read the book "Switch" by Chip and Dan Heath, and I recommend it to anyone who manages or directs a BI program. It's full of great advice for how get individuals and groups moving in new directions. To crystallize the authors' key messages (which by the way is critical to managing change), the way to get people to change habits is to appeal to their head, heart, and herd.

Appeal to the Head

Any change has to make sense to individuals and be easy to accomplish. The authors recommend establishing a goal that gives people clear direction accompanied by a few simple steps to achieve the goal. The goal motivates and the script activates.

For example, a recently deregulated Brazilian railroad company with minimal cash reserves needed to achieve financial stability quickly. To achieve the goal, the CEO established four clear rules to guide his team's immediate decisions: 1) only invest in projects that generate short-term revenue 2) apply solutions that require the least amount of upfront cash 3) a quick fix is better than a more ideal solution that takes longer to implement 4) reuse materials before you buy new ones. By "scripting the critical moves", the Brazilian company made it easy for employees to make the right decisions to restore the company to financial health.

A corollary to this maxim is to make the change process as easy as possible. For example, Amazon.com made online ordering extremely simple and quick with its "One Click" ordering. Any change initiative should strive to be as effortless and painless as that.

Appeal to the Heart

The Heath brothers say the most critical part of any change initiative is to appeal to people's emotions. Emotions, more than reason, guide our actions, attitudes and decisions. In the battle between one's head and heart (i.e. emotions, desires), the heart usually wins. As the Bible says, "the spirit is willing but the flesh is weak."

The book describes the tale of an operations manager at a manufacturing company who wanted to reduce purchasing costs by $1 billion in five years. To implement far-reaching changes required to achieve this goal, he didn't just present a detailed cost-benefit analysis. He also collected 424 different pairs of work gloves the company had bought and tagged each with a price. He then dumped this collection on a conference room table and invited division presidents to come and examine the "glove shrine." By demonstrating the wastefulness of his company in a tactile way that executives could observe and touch first hand, the operations manager triggered a major process reengineering initiative at the company.

The corollary here is to break big projects into small tasks that are easy for people to start working on. The key to any change effort is to achieve a quick win that creates momentum and confidence.

Appeal to the Herd

The final tenet of change management is to leverage group dynamics. To an astonishing degree, humans take cues about how to behave from those around them. For example, research has shown that "obesity is contagious," the authors of "Switch" write.

A research study that was not cited in the book explains that the only surefire way to get kids to eat peas is to socialize them with kids who do. Ordering kids to eat their peas doesn't work, nor does reasoning with them about the health benefits or offering them incentives. And the research shows that what's true with children is also true with adults.

To leverage group dynamics, the authors recommend creating a reform group that exhibits the new attitudes or behavior. From a BI perspective, this might involve implementing a new technology in a single department that is receptive to it and promoting the benefits they received. It can also be done by implementing dashboards that publicize the performance of peers against metrics that represent behaviors a company wants to promote. Dashboards not only get people's competitive juices flowing, they communicate the desired behavior and show people exhibiting it.

The reality today is that Hadoop is too slow to support iterative analysis. It not only runs in batch, but it's not even a terribly efficient batch environment. Hadoop version 1.0, at least, has no concept of joins, so programmers must string together hundreds of MapReduce jobs to execute relatively simple queries. There is minimal workload management so a single query will consume all the resources of a single cluster or partition.(Hadoop 2.0 addresses these and other deficiencies.)

Consequently, most companies today use Hadoop as a gigantic extraction and transformation engine that captures, stores, and processes semi-structured data and pushes the results into SQL-based environments where business users query and analyze the data using familiar SQL-based tools. Interestingly, few companies use Hadoop for other batch-oriented analytical workloads, such as scheduled reporting and data mining. The hype almost exclusively emphasizes exploration and discovery.

Real-time or Bust!

So, Hadoop's analytical mystique belies the facts. This is the primary reason that leading Hadoop vendors have rushed this year to unveil real-time query engines that run inside Hadoop. Cloudera launched Impala, EMC Greenplum announced Hawq, and Hortonworks is betting on Hive. All three not only claim to turn Hadoop into a bonafide, iterative analytical environment, they also support SQL- or SQL-like interfaces to make it easier for non-data scientists to access Hadoop data. Of course, these environments are brand new and relatively untested. The jury is out whether they truly can reinvent Hadoop in the real-time image its ardent supporters envision.

The alternative is to move the data into SQL-based analytical engines, such as Teradata or IBM Netezza, that are designed to run complex queries and analytical functions against terabytes of data. Every database vendor now offers connectors to move data from Hadoop into their proprietary environments. But these systems come with a steep pricetag and require IT administrators to move large volumes of data across thin pipes--not a smart thing to do if we're truly talking about "big data."

Consequently, early adopters of Hadoop have asked vendors to deliver a real-time query engine and they have heeded their calls. If Hadoop truly supports real-time queries, it could reinvent the entire analytical landscape and make investors in Hadoop startups fabulously wealthy. But don't rush to call your broker: Hadoop will have a longer trail to real-time nirvana than the relational database management system (RDBMS), which has been on that path for more than 30 years.

However, organizations don't need to purchase nine different tools from nine different vendors. Some large vendors offer suites of BI tools that span multiple technologies. Thus, it is possible for a company to standardize on a single BI vendor whose toolset encompasses all nine technology categories. But this may not be the best strategy since an "all-in-one" solution does not necessarily deliver "best of breed" products.

Figure 1. BI Tools Landscape and Framework

Top Down versus Bottom Up

The size of the ovals in figure 1 indicates the degree to which each technology category spans a range of characteristics in three dimensions: functionality, scope, and types of users. The horizontal dotted line refers to the demarcation between "top down" and "bottom up" BI.

Top down. The top-down world of BI delivers reports and dashboards to casual users (e.g. executives, managers, and front-line workers) who use information to do their jobs. Eighty percent of the time, these users simply want to monitor business processes and metrics for which they are held accountable. An interactive dashboard is the ideal information interface for these users because it enables them to quickly view the status of their processes and drill into more detail if needed. The top-down world is generally driven by IT which adheres to a systematic process of gathering requirements, creating data models, and delivering data warehousing-centric applications that are designed to support on-going needs.

Bottom up. In contrast, bottom-up BI is designed to meet the needs of power users (e.g. business analysts, statisticians, and data scientists) who are hired to analyze information on a daily basis. These users explore data in an ad hoc fashion and then manipulate that data with analytical algorithms to answer new questions that aren't addressed in the standard reports and dashboards of the top-down world. The bottom-up world is generally driven by business analysts who piece together data from a variety of sources, both inside and outside the organization, using low-cost analytical tools to meet immediate or pressing needs.

Two Goliaths

Out of the nine categories of BI technologies, two stand out as the most dominant: relational OLAP (ROLAP) and visual discovery. These two technologies--which come from different ends of the BI spectrum--offer greater scope and functionality and meet the needs of more types of users than the other BI technologies. ROLAP hails from the top down, IT-driven, enterprise-oriented world of data warehouses and data marts, while visual discovery technology hails from the bottom-up, analyst-driven, departmental-oriented world of spreadsheets and analytical models. These two technologies are now on a crash course to battle for supremacy in the BI market.

Visual discovery. ROLAP tools have a long history in the BI market, but visual discovery tools come from aggressive upstarts that harbor not-so-secret ambitions to dominate the BI market. Led by vendors such as Tableau, QlikTech, and Spotfire, this new category of BI tools has sparked a swift rearguard action among traditional enterprise BI vendors which have all shipped visual discovery products in the past 12 months to protect their flanks from the nimble invaders. (See my latest report, "Visual Discovery Tools: Market Segmentation and Product Positioning.")

Two years ago, the visual discovery oval was only half as big as it is today because the tools were primarily used by individual analysts for personal use and to build departmental dashboards. But because of its ease of use, low cost, and visual interactivity, visual discovery technology has spread quickly. It has already subsumed the desktop analysis category and most of the multidimensional OLAP category (except write-backs and complex dimensional calculations), and is now threatening to move into the domain of ROLAP tools as well as almost every other category of BI tools. (See figure 1.)

Self-service BI

The market for visual discovery is expanding rapidly because the technology makes triply good on the promise of self-service BI, which has been the holy grail of BI professionals for nearly 20 years. The tools not only empower business analysts to explore and analyze data without boundaries, but also create interactive dashboards for departmental colleagues. This dual function gives the tools a stealth opening into the world of top-down BI. Moreover, visual discovery dashboards empower casual users to do more with their sandboxed data than just view it: they can navigate and modify the data, and collaborate with others, among other things. So visual discovery tools pack a powerful wallop: they enable business users (not IT) to analyze data, author dashboards, and interact with dashboard data.

You would think that the visual discovery vendors would be content with their triple play, but they aren't. They are moving rapidly into enterprise BI markets long dominated by ROLAP and reporting tools, pulled by their biggest customers who want to standardize on the tools throughout their organizations. Visual discovery tools also are impinging on other adjacent BI categories, such as data mining, big data analytics, and operational reporting.

A Path to the Future

Given the market landscape described above, what should a BI vendor do to grow market share? The natural evolution of BI products is to migrate from desktop to departmental to enterprise segments; in essence, from the lower-left side of the quadrant chart to the upper right. (See figure 2.) Even ROLAP tools started as desktop products (remember MicroStrategy in the early 1990s??) and then moved up the evolutionary ladder from departmental to enterprise solutions. For better or worse, visual discovery tools are tracing the same path. (Sometimes I wish departmental BI vendors would remain content as departmental players instead of taking the risky journey into the enterprise.) Departmental vendors that make a successful transition figure out how to balance the characteristics that made them successful as departmental players with new enterprise features.

Figure 2. Market Evolution

Segmenting the market. However, it's impossible for one product to appeal to both desktop, departmental, and enterprise users. The requirements conflict. Departmental users want BI tools that are easy to use, fast, quick to deploy and low cost. Enterprise users want scalability, reliability, security, and reusability. The key is not to meet the needs of both markets in one product but to create separate products, one for each market that is packaged, branded, and priced distinctly. It's also important that vendors provide a seamless migration path from one product to the other, not just for the software, but for data and metadata as well.

Marketers need to have a clear picture of the target customer in each market and the words, features, and pricing that resonate with each. They need to remember that departmental customers buy software to satisfy an immediate need. Generally, they are business people trying to solve a specific business problem. They don't want or need to concern themselves with architecture and other long-term systems requirements. They don't care who supports them, IT or a third party, as long as they get answers to issues fast at an affordable price.

In contrast, enterprise customers buy software to address long-term needs. As such, the IT department plays a major role in the buying decision since they need to scout out whether the software aligns with the company's existing infrastructure and architectural requirements. On the business side, the buyers want to know that the vendor is a leader in the industry and has a vision for the future. They want to see the vendor "check off all the boxes" on a request for proposal for features and functions that may or may not ever need. In other words, they want the BI vendor to show how they support all nine categories of BI technology as well as all information delivery channels (e.g. mobile, cloud, Web, desktop) and the latest technologies (e.g. Hadoop, social media, and analytics). This is a tall order for departmental BI vendors who want to make the leap into the enterprise BI market because it sucks up a lot of research and development resources and creates headaches for marketers whose must create distinctive messaging for a rapidly expanding set of features, functions, and products.

At the same time, enterprise players (or enterprise wannabees) can't focus so much on the enterprise play that they forget the engine of growth, which is their departmental offering. And likewise, the desktop offering is the engine of growth for their departmental offering. But the temptation is to focus an inordinate amount of time, money, and resources on product development and marketing for the enterprise play because this segment is more profitable and ultimately constitutes the lionshare of revenue. But this is not a long-term winning strategy because it chokes off the engines of growth for each segment.

By now, most enterprise BI players have figured out how to handle this challenge. Yet they are still vulnerable to competition from visual discovery competitors because of the self-service triple play described above. Although top-down BI products provide self-service BI, the IT department needs to first create a semantic layer or populate the mashboard library with predefined objects before business users can author or interact with the software in a self-service manner. Trying to push down self-service from the top-down world is a lot harder than pushing self-service from the bottom up.

Market Vectors

Given the breadth of the BI market and its current velocity, vendors need to follow one of three market vectors, while not losing sight of the others. These vectors are operational analytics, enterprise dashboards, and big data analytics. (See figure 3.) It will be hard to dominate all three vectors, given the competition and speed of product evolution in each sector.

Figure 3. BI Vectors

1. Enterprise dashboards. This represents the sweet spot of the BI market because enterprise dashboards meet the information needs of the greatest number of business users--casual users who simply want to monitor their areas of responsibility. These dashboards can be simple, if departmental in scope, or complex, if enterprise in scope. The degree of difficulty largely has to do with the degree of data governance required to deliver trustworthy data and the challenge of delivering adequate performance as the number of users and volume of data and metrics in each dashboard increase.

2. Operational analytics. Products in this fast-growing sector monitor and analyze data in motion versus data at rest--in other words, they apply rules and calculations to real-time event streams, such as those emanating from Web and systems logs, sensor networks, digital devices, and commercial machinery. This sector sits at the intersection of analytical intelligence (e.g. data mining) and continuous intelligence (complex event processing). (See my BI Framework 2020 for a description of these intelligences.) This is a small market but one that is finally gaining traction with customers.

3. Big data analytics. Big data analytics is another hot market, fueled by interest in Hadoop and NoSQL technologies. This space is wide open right now and vendors from three distinct markets--database, BI/visualization, and analytics--are battling for hegemony. My bet is on products that combine all three capabilities in a single integrated toolset that exploits the latest advances in CPU, memory, visualization, and parallelization and can be purchased along with the hardware (e.g. appliance or cloud.) This aligns with the macro trend in the high tech industry toward convergence of hardware and software as well as operations and analytics.

Departmental strategies. Desktop and departmental BI vendors will need to align closely with one of these vectors. They simply don't have the resources to play in all of them, especially if the aspire to move into the enterprise space, which will suck up valuable resources as they beef up their metadata, security, clustering, and other enterprise capabilities. Focusing on the current leaders, it would make sense for QlikTech to track the enterprise dashboards vector, Tableau to pursue big data analytics, and Spotfire to chase operational analytics.

Enterprise strategies. On the other hand, enterprise BI players will need to play in all three sectors, but specialize in one. They will also need to offer desktop and departmental tools to round out their portfolios and provide an attractive and affordable on-ramp for future enterprise customers. Obviously, the challenge for enterprise BI vendors is how provide compelling product value and deliver clear and persuasive marketing messages while chasing all three vectors and downstream markets. This will not be easy. It will take a clever marketing officer with a deep understanding of customers, markets, and messages to thread this needle.

Summary

The BI market is complex and evolving. Among the nine distinct categories of BI technologies, two have the largest footprint in customer accounts--ROLAP and visual discovery--and both are contending for supremacy in the BI market. ROLAP is the preeminent top-down technology, while visual discovery is the preeminent bottom-up technology. Both make good on the promise of self-service BI, but visual discovery offers greater versatility by empowering both casual and power users to analyze data and author dashboards.

But no technology or product can thrive without people who understand the current dynamics of the BI market and know how to position products accordingly. Desktop and departmental vendors need to bridge the chasm between departmental and enterprise solutions while pursuing one of three emerging BI vectors. Enterprise players need to span all BI technologies and emerging vectors while retaining clear, crisp messaging for each.

APPENDIX: THE BI TOOLS FRAMEWORK

The BI Tools Framework (see figure 1) positions categories of BI tools by functionality, scope of deployment and type of intelligence (top-down versus bottom-up). To deliver a comprehensive set of BI capabilities, organizations need to implement BI products or suites that span all dimensions of the BI Tools Framework.

Here are definitions of the key terms in the framework:

Top-down intelligence: Top-down intelligence consists of reports and dashboards that answer predefined questions or monitor business processes using metrics aligned with strategic objectives. Top-down tools query data sets with predefined schema, including data warehouses and data marts. Top-down tools are installed, designed and maintained by the IT department but operated by power users and super users.

Bottom-up intelligence: Bottom-up intelligence consists of ad hoc analysis and mining tools that answer unanticipated questions arising from new and changing business conditions. Bottom-up tools query any type of data in any application or system and use flexible schema and rich visualizations to effectively manipulate and analyze data. Bottom-up tools are used by business analysts, statisticians and data scientists with minimal or no IT assistance.

Top-Down Business Intelligence Tools

Pixel-perfect reporting: Also called production or managed reporting tools, pixel-perfect reporting tools enable professional report developers to create pixel-perfect and complex reports and distribute them to large numbers of users on a scheduled basis.

Relational OLAP (ROLAP): Relational OLAP tools provide a dimensionalized view of data held in relational databases, using a combination of specialized schema and end-user metadata, that is, a semantic layer. IT developers use ROLAP tools to create interactive reports and dashboards that run against large data warehouses. Because ROLAP tools span reporting, dashboarding and analysis, they are ideal for delivering layered enterprise dashboards that provide three-click access to any data from a top-level view, assuming adequate performance.

Operational reports and dashboards: Unlike ROLAP tools, which connect to data via specialized schema and architected metadata, operational reports and dashboards tools connect directly to source systems that run the business. Also called data mashups, these tools are used to create operational or executive dashboards that monitor near-real-time business activity across multiple applications and systems. Some higher-end operational dashboards monitor real-time event streams.

Ad hoc reports and dashboards: Set up by IT professionals, these tools enable motivated business users, that is, super users, to create ad hoc reports and dashboards. Super users create reports by selecting data elements (metrics, dimensions, attributes) from a scrubbed list of data objects (semantic layer) and then format the result set. They create ad hoc dashboards ("mashboards") by dragging and dropping predefined content (charts, tables, controls) from a library onto a dashboard canvas. Ad hoc tools are often extensions of ROLAP or pixel-perfect reporting tools.

Bottom-Up Business Intelligence Tools

Desktop analysis: Epitomized by Microsoft Excel (and now PowerPivot), desktop analysis tools are designed for business analysts who want free-form SQL access to any data and the flexibility of a desktop tool to merge, integrate and model data to answer any question they need to answer.

Visual discovery: Visual discovery tools are self-service, in-memory analysis tools that enable business users to access and analyze data visually at the speed of thought with minimal or no IT assistance and then share the results of their discoveries with colleagues, usually in the form of an interactive dashboard.

Multidimensional OLAP: These tools store data in a specialized multidimensional format and either precalculate or dynamically calculate data values at the intersection of every level in the dimension hierarchy, providing fast query performance. Designed for business analysts, the tools require IT professionals to set up, design, load and manage the databases. They are constrained in size because of the explosion of data caused by the calculation of dimensional values.

Data mining workbench: These tools provide a drag-and-drop development environment for statisticians who need to design, build and manage analytical models, either individually or in a team.

Big data analytics: These are highly scalable analytical systems that run on grids of computers, including Hadoop and MPP database appliances, and contain libraries of analytical functions that enable statisticians and data scientists to explore, manipulate and model large volumes of data without having to move the data to a workstation or departmental server. They also support analytical sandboxes that enable analysts to upload private data and mix it with corporate data to conduct analyses as well as strong visualization capabilities.