Blog: Wayne Eckerson Subscribe to this blog's RSS feed!

Wayne Eckerson

Welcome to Wayne's World, my blog that illuminates the latest thinking about how to deliver insights from business data and celebrates out-of-the-box thinkers and doers in the business intelligence (BI), performance management and data warehousing (DW) fields. Tune in here if you want to keep abreast of the latest trends, techniques, and technologies in this dynamic industry.

About the author >

Wayne has been a thought leader in the business intelligence field since the early 1990s. He has conducted numerous research studies and is a noted speaker, blogger, and consultant. He is the author of two widely read books: Performance Dashboards: Measuring, Monitoring, and Managing Your Business (2005, 2010) and The Secrets of Analytical Leaders: Insights from Information Insiders (2012).

Wayne is currently director of BI Leadership Research, an education and research service run by TechTarget that provides objective, vendor neutral content to business intelligence (BI) professionals worldwide. Wayne’s consulting company, BI Leader Consulting, provides strategic planning, architectural reviews, internal workshops, and long-term mentoring to both user and vendor organizations. For many years, Wayne served as director of education and research at The Data Warehousing Institute (TDWI) where he oversaw the company’s content and training programs and chaired its BI Executive Summit. He can be reached by email at weckerson@techtarget.com.

October 2011 Archives

Business intelligence is changing. I've argued in several reports that there is no longer just one intelligence--i.e., business intelligence--but multiple intelligences, each supporting a unique architecture, design framework, end-users, and tools. But all these intelligences are still designed to help business users leverage information to make smarter decisions and support the creation of either reporting or analysis applications.

The four intelligences are:


  1. Business Intelligence. Addresses the needs of "casual users," delivering reports, dashboards, and scorecards tailored to each user's role, populated with metrics aligned with strategic objectives and powered by a classic data warehousing architecture.

  2. Analytics Intelligence. Addresses the needs of "power users," providing ad hoc access to any data inside or outside the enterprise to answer business questions that can't be identified in advance using spreadsheets, desktop databases, OLAP tools, data mining tools and visual analysis tools.

  3. Continuous Intelligence. Collects, monitors, and analyzes large volumes of fast-changing data to support operational processes. It ranges from near real-time delivery of information (i.e., hours to minutes) in a data warehouse to complex event processing and streaming systems that trigger alerts.

  4. Content Intelligence. Gives business users the ability to analyze information contained in documents, Web pages, email messages, social media sites and other unstructured content using NoSQL and semantic technology.

You may wonder how all these intelligences fit together architecturally. They do, but it's not the clean, neat architecture that you may have seen in data warehousing books of yore. Figure 1 below depicts a generalized architecture that supports the four intelligences.

Figure 1. BI Ecosystem of the Future
BI Ecosystem of Future.jpg

The top half of the diagram represents the classic top-down, data warehousing architecture that primarily delivers interactive reports and dashboards to casual users (although the streaming/complex event processing (CEP) engine is new.) The bottom half of the diagram adds new architectural elements and data sources that better accommodate the needs of business analysts and data scientists and make them full-fledged members of the corporate data environment.

A recent report I wrote describes the components of this architecture in some detail and provides market research on the adoption of analytic platforms (e.g. DW appliances and columnar and MPP databases), among other things. The report is titled: "Big Data Analytics: Profiling the Use of Analytical Platforms in User Organizations." You can download it for free at Bitpipe by clicking on the hyperlink in the previous sentence.

Since "Multiple Intelligences" framework and BI ecosystem that supports it represent what I think the future holds for BI, I'd love to get your feedback.


Posted October 21, 2011 9:35 AM
Permalink | No Comments |

I used to think that data virtualization tools were great for niche applications, such as creating a quick and dirty prototype or augmenting the data warehouse with real-time data in an operational system or accessing data outside the corporate firewall. But now I think that data virtualization is the key to creating an agile, cost-effective data management infrastructure. In fact, data architects should first design and deploy a data virtualization layer prior to building any data management or delivery artifacts.

What is Data Virtualization? Data virtualization software makes data spread across physically distinct systems appear as a set of tables in a local database. Business users, developers, and applications query this virtualized view and the software automatically generates an optimized set of queries that fetch data from remote systems, merge the disparate data on the fly, and deliver the result to users. Data virtualization software consumes virtually any type of data, including SQL, MDX, XML, Web services, and flat files and publishes the data as SQL tables or Web services. Essentially, data virtualization software turns data into a service, hiding the complexity of back-end data structures behind a standardized information interface.

With data virtualization, organizations can integrate data without physically consolidating it. In other words, they don't have to build a data warehouse or data mart to deliver an integrated view of data, which saves considerable time and money. In addition, data virtualization lets administrators swap out or redesign back-end databases and systems without affecting downstream applications.

The upshot is that IT project teams can significantly reduce the time they spend sourcing, accessing, and integrating data, which is the lionshare of work in any data warehousing project. In other words, data virtualization speeds project delivery, increases business agility, reduces costs, and improves customer satisfaction. What's not to like?

Long Time Coming. Data virtualization has had a long history. In the early days of data warehousing (~1995), it was called virtual data warehousing (VDW) and advocates positioned it as a legitimate alternative to building expensive data warehouses. However, data warehousing purists labeled VDW as "voodoo and witchcraft" and chased it from the scene. During the next 10 years, data virtualization periodically resurfaced, each time with a different moniker, including enterprise information integration or EII and data federation, but the technology never got much traction, and vendor providers came and went.

Drawbacks. One reason data virtualization failed to take root is politics. Source systems owners don't want BI tools submitting ad hoc queries against their operational databases. And these administrators have the clout to lock out applications they think will bog down system performance.

Other traditional drawbacks of data virtualization are performance, scalability, and query complexity. The engineering required to query two or more databases is complex and becomes exponentially more challenging as data volumes and query complexity grow. As a result, data virtualization tools historically have been confined to niche applications involving small volumes of clean, consistent data sets that require little to no transformation and complex joins.

Architectural Centerpiece?

Today, however, data virtualization is making a resurgence. Advances in network speeds, CPU performance, and available memory have significantly increased the performance and scalability of data virtualization tools, expanding the range of applications they can support. Moreover, data virtualization vendors continue to enhance their query optimizers to handle more complex queries and larger data volumes. Also, thanks to the popularity of data center virtualization, many organizations are open to exploring the possibility of virtualizing their data as well.

But does this mean data virtualization is ready to take center stage in your data management architecture? I think yes. Last week, I listened to data architects from Qualcomm, BP, Comcast, and Bank of America discuss their use of data virtualization tools at "Data Virtualization Day," a one-day event hosted by Composite Software, a leading data virtualization vendor. After hearing their stories, I am convinced that data virtualization is the missing layer in our data architectures.

These architects reported no performance or scalability issues with data virtualization. If they encounter a slow query, they simply persist or cache the target data in a traditional database and reconfigure the semantic layer accordingly. In other words, they virtualize everything they can and persist when they must. This approach overcomes all physical and political obstacles to data virtualization, while improving query performance and project agility. And some hard-core "data virtualizers" do away with a data warehouse altogether, preferring to persist snapshots of data that requires an historical or time-series view.

Today, the biggest obstacles to the growth of data virtualization are perceptions and time. Given the innate bias among data warehousing professionals to persist everything, most data architects doubt that data virtualization tools offer adequate performance for their query workloads. In addition, it takes time to introduce data virtualization tools into an existing data warehousing architecture. The tools must prove their worth in an initial application and build on their success. Since enterprise-caliber data virtualization tools cost several hundred thousand dollars, they need a well-respected visionary to advocate for their usage. In most organizations, it's easier to go with the flow than buck the trend.

Nonetheless, the future looks bright for data virtualization. Since most of our data environments are heterogeneous (and always will be), it just makes sense to implement a virtualization layer that presents users and applications with a unified interface to access any back-end data no matter where it's located or how its structured. A layer of abstraction that balances federation and persistence can do two things that every IT department must deliver: lower costs and quicker deployments.


Posted October 17, 2011 3:37 PM
Permalink | No Comments |

Most business intelligence (BI) professionals understand the need for the business to drive the BI program to achieve success. But, most don't have a clue how to make this happen.

To be fair, some organizations are lost causes. Their executives view BI as a reporting cost-center driven by IT. They don't understand the value of information to optimize performance and deliver a sustainable advantage. They haven't figured out that "data is the new oil" and that whoever masters the means of data production, wins.

In most other organizations, the business is well meaning, but too busy and preoccupied to commit the necessary time to ensure the success of a BI program. When push comes to shove, they still relegate the duties of delivering information-centric applications to the IT department, entrusting them to make key decisions about semantics, metrics, and targets. And most competent IT teams are happy to oblige, since they often know the business as well as or better than many in the business.

BOBI on Board

Two BI Teams. However, there are a few organizations in which executives both talk the talk and walk the walk. These executives make substantial investments in BI but do so in a unique way: they don't just fund the acquisition of technology and hire IT staff to manage it, they create and staff a business-oriented BI team to complement the IT-oriented BI staff. In other words, they support TWO BI teams, one focused on the business, the other on technology.

This business-oriented BI team doesn't yet have an official designation in the BI lexicon. It currently goes by many names: enterprise data solutions, information management, business information analysis, business insights and analytics, and even business intelligence. But since it's a business-oriented BI (BOBI) team, let's just cut to the chase and call it BOBI.

BOBI teams typically report to an executive on the business side. Ideally, it's the chief operations officer or chief technology officer not the head of a department, like finance or marketing who can limit BOBI's activities to a too narrow domain. BOBI should have an enterprise focus. Working jointly with IT, BOBI should build the proverbial data factory to ensure clean, consistent, and accurate data to power all BI solutions throughout the organization.

More than BI Governance

I'll admit, BOBI is a revelation to me. Most corporate BI teams I've seen are part of the IT department and consist mainly of technologists with a business bent. I've always advocated that the primary duty of such BI teams is to foster a BI governance structure comprised of two voluntary, ad hoc committees: a steering committee of executive level sponsors and a working committee of business analysts and subject matter experts.

The executive steering committee provides funding, prioritizes projects, and approves the high-level BI roadmap, while the BI working committee works with IT to create the roadmap, flesh out data warehouse subject areas, select tools, and prioritize enhancements. I've always said these business analysts and SMEs can be your best allies or worse enemies, so it's best to make them full partners in the BI journey.

Missing Link. However, what I missed is that these business analysts should not be part-time volunteers with other priorities and bosses; they should be allocated full-time to the BI program. In addition, they should be assigned to a dedicated BI team led by a business-savvy BI director who also has ample experience running BI and technology projects.

BI-Lingual Professionals

I thank Nick Triantos and Andre Synnett for steering me straight. Nick is currently director of enterprise BI and Data Programs at McAfee and former director of Quality Data Systems at Cisco, while Andre is vice president of the BI Competency Center and the soon-to-be chief data officer at Caisse de depot, a large pension fund investment firm in Quebec. Both come from the business but have substantial technology experience. They are the proverbial "purple people" needed to succeed with BI: neither blue from the business or red from IT, but a perfect blend (i.e., purple) of both.

Both Nick and Andre run business teams dedicated to BI that sit between the business and IT. Team members, like themselves, are bilingual ("BI-lingual"): they can speak both business and technology.

On the business side, team members gather requirements while simultaneously evangelizing the capabilities of their respective companies' BI infrastructure to address current business objectives. They also develop the BI roadmap, manage the BI budget, oversee BI and data governance programs, and create change management programs. They often establish standards for the BI user experience and oversee the BI tool selection process.

On the technical side, they translate business requirements into technical specifications, manage metadata, and document best practices for delivering BI solutions, They work with data architects to flesh out the BI roadmap, project managers to accommodate shifting user requirements, technical architects to select and deploy BI tools, and help desk staff to ensureeffective end-user support and training.

Summary

To succeed with BI, you need to convince executives to step up and fund BOBI--a
permanent business-oriented BI team. Without such an investment, the odds of achieving BI success are stacked against you.


Posted October 4, 2011 7:11 AM
Permalink | No Comments |