"It's ironic that we are the group that measures everything in the organization, but we don't have a good way to measure ourselves and our effectiveness."
This is basically what Eric Colson, former VP of Data Science and Engineering at Netflix, told me when I asked him how he measured the success of his BI and analytics team. (Eric is now chief data officer at Stitch Fix.)
When I pressed him, Eric said the best empirical measure of BI success that he could come up with was the number of times business unit heads mentioned his team at their regular operational review meetings. If a business head tells executives that he is partnering with the BI team to deliver a strategic project, Eric takes that as a sign his team has the confidence of the business and is making value-added contributions. "We must be doing something right if they keep wanting to work with us as strategic partners," says Eric.
If you are like most BI managers, measuring success is an afterthought. It's hard enough to get approval for a project and deliver results without having to kick off another project to measure your team's performance. And if you do have the time and interest, what exactly do you measure?
Usage tracking. Most BI managers track usage to gauge performance and value. They monitor how many users have BI licenses, how often they log in, how many reports they run on average, how many queries they run against which data elements, and so on. But high usage doesn't necessarily mean users are getting a lot of value or that this value is commensurate with the organization's investments in BI. For example, although you might have 1,000 users, perhaps only 25% log in weekly, and when they do, they only run one report each, which they look at for just five minutes. So, there is a lot of activity, but very little uptake.
Surveys. Some more ambitious BI managers send surveys to BI users to gauge their satisfaction with the BI tools and reports. Unfortunately, from what I've seen, the response rate to these surveys is pretty dismal (but that's true for all surveys these days) which means results are potentially skewed. Typically, you only hear from those who are really happy, frustrated, or disappointed. Unfortunately, the real value comes from the great unwashed masses who didn't respond. In my mind, this undermines the value of a survey as a measuring stick of success.
Social media analysis. One BI manager I know wants to add social media features to his team's BI reports, such as giving users the ability to rate, comment, and share reports with peers. Then using social media analytics, he can evaluate the value of each report, using both empirical and subjective data, and by extension, the value users get from BI deliverables. This also helps the BI team delete unused and undervalued reports and get a better sense of what data in what format users find helpful.
Spreadmarts. In the past, I've jokingly said that a BI program's success is inversely proportional to the number of spreadmarts in its environment. In theory, the fewer renegade data shadow systems that exist in a company, the more likely that users are getting value from the BI team's reports, dashboards and self-service reporting tools. Of course, this means a BI manager has to find and monitor all the spreadmarts in her organization. But this is like playing whack-a-mole. As soon as she discovers one spreadmart and consolidates it into the data warehouse, three more spring up that she isn't aware of.
Cost efficiencies. The best BI managers track the costs of making decisions. Before they initiate a BI project, they establish a baseline set of figures that take into account the cost of hardware and software licenses and the number of hours per week that analysts spend accessing data rather than analyzing it multiplied by their fully loaded hourly salaries. After completing a BI project, these BI managers measure these items again and compare the results to the baseline to gauge the financial lift of the BI project.
Companies that implement BI for the first time can usually wring lots of costs from their decision making processes by making the data acquisition and delivery process more efficient. But companies with mature BI programs don't have this luxury because they've already streamlined BI processes. BI managers here must justify continued investment in the BI program on the basis of the program's strategic value to the organization. This usually involves measuring the value of better decisions, mission-critical processes powered by data, or more informed workers. This is not easy to do, but it can be done. Unfortunately, the results can always be disputed by a cynical executive.
Full circle. And this brings us back to Eric Colson. Perhaps tracking the number of mentions the BI team gets in an executive meeting is not an exact science. And perhaps it's a bit unseemly or ego maniacal (and Eric doesn't do this.) But as far as I can tell, it's the best metric we have for truly measuring the value of a BI program.
Let me know what techniques you've used to measure BI success. (And you can learn more about Eric Colson and other top BI leaders in my new book, "Secrets of Analytical Leaders: Insights from Information Insiders" available at Amazon.com.)
Posted January 30, 2013 3:13 PM
Permalink | No Comments |