Blog: Wayne Eckerson Subscribe to this blog's RSS feed!

Wayne Eckerson

Welcome to Wayne's World, my blog that illuminates the latest thinking about how to deliver insights from business data and celebrates out-of-the-box thinkers and doers in the business intelligence (BI), performance management and data warehousing (DW) fields. Tune in here if you want to keep abreast of the latest trends, techniques, and technologies in this dynamic industry.

About the author >

Wayne has been a thought leader in the business intelligence field since the early 1990s. He has conducted numerous research studies and is a noted speaker, blogger, and consultant. He is the author of two widely read books: Performance Dashboards: Measuring, Monitoring, and Managing Your Business (2005, 2010) and The Secrets of Analytical Leaders: Insights from Information Insiders (2012).

Wayne is currently director of BI Leadership Research, an education and research service run by TechTarget that provides objective, vendor neutral content to business intelligence (BI) professionals worldwide. Wayne’s consulting company, BI Leader Consulting, provides strategic planning, architectural reviews, internal workshops, and long-term mentoring to both user and vendor organizations. For many years, Wayne served as director of education and research at The Data Warehousing Institute (TDWI) where he oversaw the company’s content and training programs and chaired its BI Executive Summit. He can be reached by email at weckerson@techtarget.com.

March 2011 Archives

iStock_000011624240XSmall.jpg

One of the key challenges in business intelligence (BI) is balancing corporate and departmental interests. This is a thread that weaves through most discussions of BI governance, data governance, and IT governance.

The tension between corporate standards and departmental autonomy goes well beyond the domain of BI and even information technology. In fact, it's endemic to the human condition, a driver of human history. In the United States, we fought a civil war over the scope of states' powers within a federal government. Closer to home, most of you have probably felt the heavy hand of a new corporate parent after an acquisition and have had to adjust your expectations and work habits accordingly.

Wayne, the Renegade

Ironically, I write articles like these to help corporate BI teams rein in renegade BI users, yet by nature, I'm one of those renegades. I started at TDWI when it was a private, almost family-run operation. But by the time I left, I had lived through two corporate parents, one benign and ineffective, the other bold and bossy. Given where I started with TDWI, I must admit that I bristled at taking instructions from corporate headquarters--almost on principle--especially when those standardized, sanitized instructions made little sense for our domain or cost us time, money, or customer goodwill. My mantra was "think local and resist global." I wanted complete control over my own domain. In short, I was a BI manager's worst nightmare.

Pondering my role as a potential BI "ne'er do well" got me thinking: how would I, as a corporate BI chieftain, deal with me, the proverbial BI bad boy? How would I get Wayne, the rebel, to willingly adopt corporate standards and tools and stop rushing into projects with short-term, duplicate solutions? How would I get Wayne to wait a little longer for applications that are more aligned, scalable, and durable than what he can build himself to serve his own parochial interests? How would I get him to trust the corporate BI team and rely on us instead of fight us?

Viewed through this lens, the tactics to rein in a BI renegade became clear to me. And I validated them through gut feel: if I felt they might work on me, then perhaps they might work for you in the BI trenches.

Some are bottom-up tactics that you, as a BI manager, can employ directly, such as appealing to my ego and self interest or just plain impressing the hell out of me with what you can deliver. Or you can rely on top-down tactics in which you need to recruit the CEO to buy goodwill through monetary incentives or force compliance with threat of dismissal. I've discussed many of these tactics in prior blogs, but here is a quick synthesis:

Bottom Up Strategies - What the BI Manager Can Do-

Impress me. Show me (Wayne the Renegade) that you can deliver what I want on time and within budget. Trust is hard to build, easy to destroy. Of course, since I never really know what I want until I see it, and what I need changes monthly based on ever-changing business conditions, good luck! Ideally, you know the business as well or better than I do because of your long tenure at the company and can anticipate what I want before I ask. Or you have skilled programmers who can build solutions incredibly fast without a lot of committee meetings or the need to rigidly adhere to architectural standards. In some cases, you recognize that it's more important to go fast than get it right and are willing to reset your architecture once a year to get things back in sync. Bottom line, if I think you can move faster than I can with my own resources, then you are my man (or woman)!

Give me a title. Give me a nice, ego-fattening title to entice me to run the BI working committee. Tell me you want me to help shape the BI program to address my needs. Ask me to recruit my buddies in other departments to serve on the committee with me. Repent of your team's perennial shortcomings ("too slow," "too costly," and "too structured.") Acknowledge that you want to do better, but emphasize that you can't do it without my help. I will likely be flattered by your blandishments. And once I chair the committee, I'll be unwittingly sucked into a position of protecting global interests ("my buddies") not just my own.

Give me a job. If I'm too busy to participate on a working committee or too skeptical of corporate BI, then up the ante. Recruit me to work on your team or perhaps even run it. Yes, make me your boss and give me control of your budget and people. Successful BI teams recruit people from the business, who know the business, and can talk the language of business. And if they are renegade spreadmart users, they know enough technology to pick up the rest. These people build bridges of trust between the business and IT, paying considerable dividends.

Sell me. If you can't recruit me to join the steering committee or your team, sell me on the benefits of rebuilding my application. Find out how I'd like to improve my spreadmart--e.g., add more data sources, make it real time, add a better visual interface--and the create a proposal that does that. And if I like it the way it is, show me a comparable application at a competitor that shows me the untapped potential of my application. Recruit vendors to deliver a slick demo and maybe donate some free software. Calculate how much time and money you'll save me once the application is built and trumpet how good it will make me look in the eyes of my boss.

Co-opt me. If I really don't trust you or your team, then give me access to your platform. Let me use your ETL tools and create my own data marts on your data warehouse platform. Let me add my own data to the data marts and use my own BI tools to access the data. Provide basic guidelines for using the ETL tools and platform, such as naming conventions, scheduling, and error management, and insist that I use corporate definitions for all shared data elements.

Bribe me. If all else fails and I'm a key player with sizable political clout whose support could benefit the BI program, then bribe me. Give me a freebie under the table. Take me out to lunch and learn about my mission-critical spreadmart which executives rely upon to make key decisions. Then make a secret pact in which you build out my application free of charge--or at least an initial module as a proof point--and I agree to fund the development and maintenance if what you develop meets my requirements and timeframe. Tell me you will assign your best developer to my project--someone I like and trust and who has the collective skill sets to build the entire application by himself. If you can deliver such a quick win, you'll make a powerful ally for life.

Top Down Strategies - Enacted by the CEO-

Force me. The quickest way to get buy-in to an enterprise BI program is to have the CEO require everyone in the organization to adhere to corporate standards in the use of data and reports. An edict is needed to set the stage, but won't be sufficient to stamp out renegade activities. To accomplish that, the CEO needs to "reassign" people who spend the majority of their time creating spreadmarts. However, before he restructures the workforce, he should make sure there is a viable enterprise BI alternative to pick up the slack. Unfortunately, over time, spreadmarts will creep back in to the organization unless the corporate BI team can get ahead of requirements and data sources.

Incent me. People go where the money is. So the CEO should base the bonus of a few senior managers in part on the growth of the company. The CEO should require them to serve on a cross-functional steering committee that oversees the data warehouse and BI functions. By paying them to think global, they will more effectively align their groups with global mandates to support an enterprise data warehouse and BI service. They will more effectively restrict or curtail unauthorized usage in their own departments, understanding the value that will accrue to the organization (and themselves) by adhering to a single version of enterprise data.

Summary

To rein in renegade BI users like me, you need a combination of tactics above. I can vouch that they will have some effect, if not work outright. Bottom-up strategies are things you can work on now, while top-down tactics require the cooperation of an enlightened senior executive. Nothing is a surefire bet, but if you use common sense and business savvy, you'll find common ground with even the most recalcitrant user, like me.


Posted March 25, 2011 1:49 PM
Permalink | No Comments |

In a recent blog ("What's in a Word: The Evolution of BI Semantics"), I discussed the evolution of BI semantics and end-user approaches to business intelligence. In this blog, I will focus on technology evolution and vendor messaging.

Four Market Segments. The BI market is comprised of four sub-markets that have experienced rapid change and growth since the 1990s: BI tools, data integration tools, database management systems (DBMS), and hardware platform. (See bottom half of figure 1.)

Thumbnail image for Thumbnail image for BI Market Evolution.jpg

Compute Platform. BI technologies in these market segments run on a compute infrastructure (i.e., the diagonal line in figure 1) that has changed dramatically over the years, evolving from mainframes and mini-computers in the 1980s and client/server in the 1990s to the Web and Web services in the early 2000s. Today, we see the advent of mobile devices and cloud-based platforms. Each change in the underlying compute platform has created opportunities for upstarts with new technology to grab market share and forced incumbents to respond in kind or acquire the upstarts. With an endless wave of new companies pushing innovative new technologies, the BI market has been one of the most dynamic in the software industry during the past 20 years.

BI Tools. Prior to 1990, companies built reports using 3GL and 4GL reporting languages, such as Focus and Ramis. In the 1990s, vendors began selling desktop or client/server tools that enabled business users to create their own reports and analyses. The prominent BI tools were Windows-based OLAP, ad hoc query, and ad hoc reporting tools, and, of course, Excel, which still is the most prevalent reporting and analysis tool in the market today.

In the 2000s, BI vendors "rediscovered" reporting, having been enraptured with analysis tools in the 1990s. They learned the hard way that only a fraction of users want to analyze data and the real market for BI lies in delivering reports, and subsequently, dashboards, which are essentially visual exception reports. Today, vendors have moved to the next wave of BI, which is predictive analytics, while offering support for new channels of delivery (mobile and cloud.) In the next five years, I believe BI search will become an integral part of a BI portfolio, since it provides a super easy interface for casual users to submit ad hoc queries and navigate data without boundaries.

BI Vendor Messaging. In the 1990s, vendors competed by bundling together multiple types of BI tools (reporting, OLAP, query) into a single "BI Suite." A few years later, they began touting "BI Platforms" in which once distinct BI tools in a suite became modules within a unified BI architecture that all use the same query engine, charting engine, user interface, metadata, administration, security model, and application programming interface. In the late 1990s, Microsoft launched the movement towards low-cost BI tools geared to the mid-market when it bundled its BI and ETL tools in SQL Server at no extra charge. Today, a host of low-cost BI vendors, including open source BI tools, cloud-BI tools, and in-memory visual analysis tools have helped bring BI to the mid-market and lower the costs of departmental BI initiatives.

Today, BI tools have become easier to use and tailored to a range of information consumption styles (i.e., viewer interactor, lightweight author, professional author). Consequently, the watchword is now "self-service BI" where business users meet their own information requirements rather than relying on BI professionals or power users to build reports on their behalf. Going forward, BI tools vendors will begin talking about "embedded BI" in which analytics (e.g. charts, tables, models) are embedded in operational applications and mission-critical business processes.
Data Integration Tools. In the data integration market, Informatica and Ascential Software (now IBM) led the charge towards the use of extract, transform, and load (ETL) engines to replace hand-coded programs that move data from source systems to a data warehouse. The engine approach proved superior to coding because its graphical interface meant you didn't have to be a hard-core programmer to write ETL code and, more importantly, it captured metadata in a repository instead of burying it in code.

But vendors soon discovered that ETL tools are only one piece of the data integration puzzle and, following the lead of their BI brethren, moved to create data integration "suites" consisting of data quality, data profiling, master data management, and data federation tools. Soon, these suites turned into data integration "platforms" running on a common architecture. Today, the focus is on using data federation tools to "virtualize" data sources behind a common data services interface and cloud-based data integration tools to migrate data from on premises to the cloud and back again. Also, data integration vendors are making their tools easier to use, thanks in large part to cloud-based initiatives, which now has them evangelizing the notion of "self-service data integration" in which business analysts, not IT developers, build data integration scripts.

DBMS Engines and Hardware. Throughout the 1990s and early 2000s, the database and hardware markets were sleepy tidewaters in the BI market, despite the fact that they consumed a good portion of BI budgets. True, database vendors had added cubing, aggregate aware optimizers, and various types of indexes to speed query performance, but that was the extent of the innovation.

But in the early 2000s, as data warehouse data volumes began to exceed the terabyte mark and query complexity grew, many data warehouses hit the proverbial wall. Meanwhile, Moore's law continued to make dramatic strides in the price-performance of processing, storage, and memory, and soon a few database entrepreneurs spotted an opportunity to overhaul the underlying BI compute infrastructure.

Netezza opened the flood gates in 2002 with the first data warehousing appliance (unless you count Teradata back in the 1980s!) that soon gained a bevy of imitators, offering orders of magnitude better query performance for a fraction of the cost. These new systems offer innovative new storage-level filtering, column-based compression and storage, massively parallel processing architecture, expanded use of memory-based caches, and in some cases, use of solid state disk, to bolster performance and availability for analytic workloads. Today, these "analytic platforms" are turbo-charging BI deployments, and in many cases, enabling BI professionals to deliver solutions that weren't possible before.

As proof of the power of these new purpose-built analytical systems, the biggest vendors in high-tech have invaded the market, picking off leading pureplays before they've even fully germinated. In the past nine months, Microsoft, IBM, Hewlett Packard, Teradata, SAP, and EMC purchased analytic platform vendors, while Oracle built its own with hardware from Sun Microsystems, which it acquired in 2009. (See "Jockeying for Position in the Analytic Platform Market.")

Mainstream Market. When viewed as a whole, the BI market has clearly emerged from an early adopter phase to the early mainstream. The watershed mark was 2007 when the biggest software vendors in the world--Oracle, SAP, and IBM--acquired the leading BI vendors--Hyperion, Business Objects, and Cognos respectively. Also, the plethora of advertisements about BI capabilities that appear on television (e.g., IBM's Smarter Planet campaign) and major consumer magazines (e.g. SAP and SAS Institute ads) reinforce the maturity of BI as a mainstream market. BI is now front and center on the radar screen of most CIOs, if not CEOs, who want to better leverage information to make smarter decisions and gain a lasting competitive advantage.

The Future. At this point, some might wonder if there is much headroom left in the BI market. The last 20 years have witnessed a dizzying array of technology innovations, products, and methodologies. It can't continue at this pace, right? Yes and no. The BI market has surprised us in the past. Even in recent years as the BI market consolidated--with big software vendors acquiring nimble innovators--we've seen a tremendous explosion of innovation. BI entrepreneurs see a host of opportunities, from better self-service BI tools that are more visual and intuitive to use to mobile and cloud-based BI offerings that are faster, better, and cheaper than current offerings. Search vendors are making a play for BI as well as platform vendors that promise data center scalability and availability for increasingly mission-critical BI loads. And we still need better tools and approaches for querying and analyzing unstructured content (e.g., documents, email, clickstream data, Web pages) and deliver data faster as our businesses increasingly compete on velocity and as our data volumes become too large to fit inside shrinking batch windows.

Next week, Beye Research will publish a report of mine that describes a new BI Delivery Framework for the next ten years. In that report, I describe a future state BI environment that contains not just one intelligence (i.e., business intelligence) but four intelligences (e.g. analytic, continuous, and content intelligence) that BI organizations will need to support or interoperate with in the near future. Stay tuned!


Posted March 18, 2011 2:29 PM
Permalink | No Comments |

MP900399514.JPG

I don't think I've ever seen a market consolidate as fast the analytic platform market.

By definition, an analytic platform is an integrated hardware and software data management system geared to query processing and analytics that offers dramatically higher price-performance than general purpose systems. After talking with numerous customers of these systems, I am convinced they represent game-changing technology. As such, major database vendors have been tripping over themselves to gain the upper hand in this multi-billion dollar market.

Rapid Fire Acquisitions. Microsoft made the first move when it purchased Datallegro in July, 2008. But it's taken two years for Microsoft to port the technology to Windows and SQL Server so, ironically, it finds itself trailing the leaders. Last May, SAP acquired Sybase, largely for its mobile technology, but also for its Sybase IQ analytic platform, which has long been been the leading column-store database on the market and has done especially well in financial services. And SAP is sparking tremendous interest within its installed base for HANA, an in-memory appliance designed to accelerate query performance of SAP BW and other analytic applications.

Two months after SAP acquired Sybase, EMC snapped up massively parallel processing (MPP) database, Greenplum, and reportedly has done an excellent job executing new deals. Two months later, in September, 2010, IBM purchased the leading pureplay, Netezza, in an all cash deal worth $1.8 billion that could be a boon to Netezza if IBM can clearly differentiate between its multiple data warehousing offerings and execute well in the field.

And last month, Hewlett Packard, whose NeoView analytic platform died ingloriously last fall, scooped up Vertica, a market leading columnar database with many interesting scalability and availability features. And finally, Teradata this week announced it was purchasing AsterData, a MPP shared nothing database with rich SQL MapReduce functions that can perform deep analytics on both structured and unstructured data.

So, in the past nine months, the world's biggest high tech companies purchased five of the leading, pureplay analytic platforms. This rapid pace of consolidation is dizzying!

Consolidation Drivers

Fear and Loathing. Part of this consolidation frenzy is driven by fear. Namely, fear of being left out of the market. And perhaps fear of Oracle, whose own analytic platform, Exadata, has gathered significant market momentum, knocking unsuspecting rivals on their heels. Although pricey, Exadata not only fuels game-changing analytic performance, it now also supports transaction applications--a one-stop database engine that competitors may have difficulty derailing (unless Oracle shoots itself in the foot with uncompromising terms for licensing, maintenance, and proofs of concept.)

Core Competencies. These analytic platform vendors are now carving out market niches where they can outshine the rest. For Oracle, it's a high-performance, hybrid analytic/transaction system; SAP touts its in-memory acceleration (HANA) and a mature columnar database that supports real-time analytics and complex event processing; EMC Greenplum targets complex analytics against petabytes of data; Aster Data focuses on analytic applications in which SQL MapReduce is an advantage; Teradata touts its mixed workload management capabilities and workload-specific analytic appliances; IBM Netezza focuses on simplicity, fast deployments, and quick ROI; Vertica trumpets its scalability, reliability, and availability now that other vendors have added columnar storage and processing capabilities; Microsoft is pitching is PDW along with a series of data mart appliances and a BI appliance.

Pureplays Looking for Cover. The rush of acquisitions leaves a number of viable pureplays out in the cold. Without a big partner, these vendors will need to clearly articulate their positioning and work hard to gain beachheads within customer accounts. ParAccel, for example, is eyeing Fortune 100 companies with complex analytic requirements, targeting financial services where it says Sybase IQ is easy pickings. Dataupia is seeking cover in companies that have tens to hundreds of petabytes to query and store. Kognitio likes its chances with flexible cloud-based offerings that customers can bring inhouse if desired. InfoBright is targeting the open source MySQL market, while Sand Technology touts its columnar compression, data mart synchronization, and text parsing capabilities. Ingres is pursuing the open source data warehousing market, and its new Vectorwise technology makes it a formidable in-memory analytics processing platform.

Despite the rapid consolidation of the analytic platforms market, there is still obviously lots of choice left for customers eager to cash in on the benefits of purpose-built analytical machines that deliver dramatically higher price-performance than database management systems of the past. Although the action was fast and furious in 2010, the race has only just begun. So, fasten your seat belts as players jockey for position in the sprint to the finish.


Posted March 8, 2011 8:20 AM
Permalink | 2 Comments |