We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Blog: Wayne Eckerson Subscribe to this blog's RSS feed!

Wayne Eckerson

Welcome to Wayne's World, my blog that illuminates the latest thinking about how to deliver insights from business data and celebrates out-of-the-box thinkers and doers in the business intelligence (BI), performance management and data warehousing (DW) fields. Tune in here if you want to keep abreast of the latest trends, techniques, and technologies in this dynamic industry.

About the author >

Wayne has been a thought leader in the business intelligence field since the early 1990s. He has conducted numerous research studies and is a noted speaker, blogger, and consultant. He is the author of two widely read books: Performance Dashboards: Measuring, Monitoring, and Managing Your Business (2005, 2010) and The Secrets of Analytical Leaders: Insights from Information Insiders (2012).

Wayne is founder and principal consultant at Eckerson Group,a research and consulting company focused on business intelligence, analytics and big data.

Editor's note: This is part II in a multi-part series on analytics.

Perhaps your executives have read the articles and books that testify to the transformative power of analytics. Or maybe they have been impressed by IBM's "Smarter Planet" television ads that provide concrete examples of how companies can harness information to make smarter, faster decisions that dramatically improve operations and outcomes. As a result, your executives want to do analytics, and they've asked you to lead the initiative.

Your first question might be: Where do we start? Or, more specifically, where does it make sense to apply analytics in our organization?

To answer this question, the first step is to define analytics. Assuming that an organization already has a data warehouse and reporting and lightweight analysis tools, analytics refers to the use of machine-learning tools to surface trends and patterns in large volumes of data. (See "What is Analytics?") Some call this class of tools data mining, predictive analytics, optimization, or advanced analytics. For the purpose of this article, I will use the term "advanced analytics" to describe these tools and techniques.

The second, and most important step to answer the question, is to recognize that there are three main reasons why organizations implement advanced analytics: 1) big data 2) big constraints and 3) big opportunities. Let's address each driver.

Big Data

If you have small amounts of data, you don't need sophisticated machine learning tools and algorithms to identify patterns, associations, trends, and outliers in the data. You can probably eyeball relevant trends by applying simple statistical functions (e.g., min/max, mean, and median) or graphing the data as histograms or simple charts. Taking it one step further, you might want to dimensionalize the data and use OLAP tools or in-memory visual analysis tools to navigate across dimensions and down hierarchies using various grids, graphs, and filters. All these techniques are largely deductive in nature--you first need to know where and how to look before you can find relevant trends and patterns.

But with massive amounts of data, just hunting for patterns using ad hoc analytical tools may prove fruitless or become too unwieldy. Plus, once you detect a pattern, you have no way of modeling it for reuse in other applications. This is where advanced analytical tools shine: you can give them a problem to solve and point them at a large data set; they then discover the patterns and relationships in the data, which they express as mathematical equations. You can use these equations to make strategic decisions or score new records to support just-in-time actions, such as online cross-sell offers, hourly sales forecasts, or event-driven maintenance.

How Big is Big? Although it doesn't make sense to apply advanced analytics to small data sets, it's not the volume of data that ultimately counts; it's the complexity of data.

For example, you probably don't need advanced analytics to analyze a terabyte of data that contains just two fields; all you really need is a simple calculation and a lot of horsepower. In contrast, a much smaller data set with hundreds of fields makes a much better candidate for advanced analytics. The tools' algorithms calculate the relationships among all these fields, which is nearly impossible to do with traditional reporting and analysis tools. These small data sets are often created by merging together data from dozens of different systems into a wide flat table desired by analytical modelers.

Big Constraints

Although advanced analytics helps when examining big or complex data, it's even more valuable as a method for overcoming internal constraints that prevent you from optimizing a business process. Advanced analytics can help fill the gap when you don't have enough time, money, or people to achieve success. When facing such constraints, advanced analytics can optimize or automate data-intensive processes.

For instance, a social services agency wants to decrease the number of clients affected by delinquent child support payments but it only has two social workers to call 5,000 deadbeat Dads. The agency uses advanced analytics to rank the targeted fathers by their propensity to pay if they receive a call from a social worker. Here, advanced analytics overcomes a labor constraint.

Another common constraint is money. For example, a retailer has $1 million dollars to spend on a direct mail campaign, which means it can only send its new catalog to 100,000 of its 500,000 customers. It uses advanced analytics to rank customers by their propensity purchase an item from the new catalog so it can optimize the uplift of its campaign.

Time can also be a constraint. For example, a company that leases rail cars must fix them when they break. The longer the company takes to fix the rail cars, the more money it loses and the less satisfied customers become. But deciding which repair shop to send the railcars requires considering many variables, including distances to various repair shops, the current wait time at each shop, the expertise at each shop, distance to the exit destination, additional problems that should be fixed while the railcar is in the shop, and so on. An online application that embeds dvanced analytic can consider all these variables and issue a recommendation to the dispatcher while he is still on the phone with the customer who called in the repair.

Another common constraint is lack of management oversight. For instance, a bank wants to standardize how it evaluates and approves loans across its branches. It uses advanced analytics to evaluate each loan and generate an automated recommendation for loan officers. In the same way, a Web site can use advanced analytics to generate personalized cross-sell recommendations to every customer, based on their past purchases and what other customers like them have purchased.

In short, organizations use advanced analytics to overcome built-in constraints that prevent them from optimizing data-intensive business processes.

Big Opportunity

Finally, it makes sense to apply advanced analytics when the business upside justifies the cost. Analytics requires hiring experts who have a strong working knowledge of statistics and know how to create analytical models. They also must be conversant in the business process that the organization wants to optimize and the data that supports that process. Obviously, these individuals aren't inexpensive. And the tools to support the modeling process and the hardware they run on cost money as well. So, before you undertake an analytics project, make sure that the business value justifies the upfront

Fortunately, the cost of building analytical models is declining. A decade ago, you had to hire a PhD statistician who could also write C code or SQL to create analytical models. Today, that is not necessarily true. A good business analyst with some data mining training can create a majority of the analytical models that organizations might need.

However, you still need PhD statisticians when models must be continuously updated, the degree of model accuracy has a huge impact on costs or profits, or the core business runs on analytical models. For example, PhD statisticians are often used to create analytical models for credit card marketing campaigns, fraud detection, and government intelligence.

Costs? How much does it cost to set up an analytical center of excellence? Assuming you hire a handful of analysts and purchase the requisite software and hardware, it's likely to cost about $1 million a year at a minimum. Many companies start smaller by exploiting open source data mining tools and data mining extensions to BI tools and databases. They also might send a talented analyst to training and give him a one-time project to test the approach. If the project succeeds, the organization makes a bigger, more permanent investment in the people and technology. Or they may hire a consultancy to run the initial project and train internal analysts in the tools and techniques.


It's best to apply analytics to data-intensive business processes that are sub-optimized due to built-in constraints, such as lack of time, people, money, and oversight. Also, advanced analytics only makes sense when the business upside is big enough and the data complex enough to justify the costs.

Posted November 5, 2011 5:04 PM
Permalink | No Comments |

Leave a comment

Search this blog
Categories ›
Archives ›
Recent Entries ›