Business Intelligence Network - The Vision for BI and Beyond

Standards and KPIs for Telecommunications Business Intelligence

Published: August 28, 2007

Individual telecommunications organizations should actively develop their own performance metrics based on the unique conditions of their business and overall goals.

If you like statistics and you like baseball, the month of August is for you! The baseball season is getting to the point where you can reasonably talk about the playoffs and the wildcard race (sorry devil rays fans… you were out of the playoffs in april/may… ). In addition, there is enough of a “sample” for Sabr-metricians to test the validity of their statistical theories, models and metrics. Here are some of the fundamental statistics that have been around since the start of the game:

• Earned run average (ERA)
  
  
• Batting average
  
  
• On base percentage

Over the course of the past 30 years, Bill James and the Sabr-metricians have built up greater and greater ideas about metrics that relate to the performance of baseball players and baseball organizations based on those fundamental metrics.  Here are some of those “new” metrics:

• On-base plus slugging (OPS)
  
  
• Walks plus hits per inning pitched (WHIP)
  
  
• Park effect

The reason that I bring this up is that recently I had the opportunity to perform a deep dive on some KPIs relating to the TeleManagement Forum’s Business Metrics Development Project  (BusMet) efforts. These KPIs are in support of the TeleManagement Forum’s enhanced Telecom Operations Map (eTOM). And, I found them to be an interesting effort in trying to do what Bill James and Sabr-metricans did for baseball.

Measuring Fundamentals

Each sports-related KPI represents a core fundamental of the sport and how the performance of a player compares to “industry” standards, peers or historical averages. For example, the “industry standard” for earned run average in baseball is 3.00. If an individual pitcher or a team can attain a 3.00 ERA, that individual or the team is considered to be good. To the uninitiated fan, this might seem like an arbitrary number. But to get a 3.00 ERA, that means you only give up one run every three innings pitched.

While I am not sure who “invented” the 3.00 ERA, it has evolved into a standard for the baseball “industry” and represents many of the key features of a KPI:

1. It is easy to calculate.
   
   
2. It is easy to communicate.
   
   
3. It is based on strong performance criteria.

The TeleManagement Forum is attempting to build similar KPIs with their BusMet efforts. While none of these will ever be confused with the simplicity of the 3.00 ERA, the KPIs are intended to develop strong industry standards and methods for benchmarking performance across many different domains that relate directly to the operation of a telecommunications organization:

• Revenue and margin
  
  
• Customer experience
  
  
• Operational efficiency

“Other than that Mrs. Lincoln…”

The BusMet effort is a great way to take some of the best minds of the telecommunications industry and have them help fellow providers with common issues. I firmly believe that it is good to have insight from many different perspectives when looking at an issue. However, since there is no such thing as a free lunch, the benefits of an industry association’s efforts have a cost (and who didn’t see the “yeah but…” coming …).

I think that standards organizations and industry associations are good at providing lobbying efforts (see the CTIA) or organizing trade shows and educational conferences (see the GSM). I am not so sure if they are good at setting up timely operational metrics and specific benchmarks for an industry.

In effect, BusMet is attempting to emulate more than one-hundred years of practice, testing and validation of baseball statistics, but just do it faster and in a committee setting. It reminds me of the observation someone once made about the platypus:

“The platypus is an animal designed by committee.”

And the platypus is a fine, fine animal … don’t get me wrong. However, it is definitely a niche player in the animal kingdom:

• Mammal that lays eggs
  
  
• A cross between a beaver and a duck
  
  
• Only lives in remote areas of Australia

Also, as with many standards bodies, the amount of time it usually takes to find consensus for a particular initiative bypasses the usefulness of the effort. I point to the CORBA and the IPDR standards. Both are great efforts and well worth the work that was put in. However, many commercial implementations of object brokers and xDR formats could not wait for the “committees” to schedule the proper committee meeting and draft a resolution on the adoption of the resolution. The speed of business requires faster time frames.

Group Think

Another issue that I have with the BusMet KPIs or any effort where an entire industry focuses on one set of best practices is that, fundamentally, competitive advantage is being driven out of that industry. In 2005, I noted a McKinsey article that suggested when groups adopt a single set of best practices, invariably they start using the same “playbook.” This type of mentality leads to the commoditization of the products and services in that industry.  It also leads to the loss of what makes each organization in that particular industry unique.

The book Moneyball has been held up by many in the sports and business intelligence industries as an example of how data and data analysis can support the success of an organization. However, following the success of that book, many people took the metrics in Moneyball to be the “gospel.” Pundits and laymen used the metrics for analysis of baseball, drafting of fantasy league teams, etc. Unfortunately, when people did this, they were all looking at the same elements of the game and, in effect, using the same playbook. They placed the same value on players and tended to compete, overly so, for those players ranked at the top – something that Sabr-metrics are designed to avoid.

The message that I think was lost in Moneyball was the fact that Billy Beane, general manager of the Oakland Athletics, was using the techniques of data analysis to find situational, yet fundamental, metrics that would allow him to gain a competitive advantage. Beane was not using a set of static metrics to run his team. He was using his analysis to determine the best metrics that allowed the A’s to identify undervalued aspects of player performance. His goal was not so much to battle with the Yankees on price, but to battle them on smarts. In this, the A’s found their niche as a VERY competitive, low budget baseball team.

Conclusion

As I look at the BusMet KPIs, I see an industry that is looking to do good work.  However, the almost academic process that relates to standards bodies should be taken into account before an organization takes the BusMet metrics to heart.

Telecommunication business intelligence organizations should look to outside influences, like the TeleManagement Forum, to benchmark their company’s operational performance and to understand their place in the overall industry. One company is not an “island.” But, the business intelligence organization should and must be mindful of how they use those KPIs. BusMet and other initiatives provide a wonderful framework for benchmarking activities. But, individual telecommunications organizations should actively and continuously develop their own set of unique performance metrics based on the unique conditions of the business, service/products/network and overall goals of the organization.

The foundation that the BusMet KPIs provide will enable the business intelligence organization to make strong decisions on what operational data they measure and how they measure it. However, guidance should be provided to avoid falling into the trap of merely managing to the “standard” metrics and not looking outside the box.

If you found this article helpful and would like to receive the latest insights each week from the experts featured on the Business Intelligence Network, please subscribe to the Business Intelligence Network Newsletter.