In my last post about BI teams executing faster, Faster BI Execution: Four Practical Approaches to Improving BI System Quality, I looked at how quality impacts BI team performance, in this fourth post in the series I examine how slow BI systems can slow the performance of BI teams.

Do users complain about how slow the system is? Do you have to create a summary table for many new reports? Do you test on sample data or only run tests once or twice because things are slow? Do developers spend a lot of time clicking refresh waiting for reports and ETL to finish?

Investing in system performance can make your business intelligence team faster. Faster systems will help BI teams deliver on shorter deadlines, adapt to changing business requirements faster, and build more functionality with few resources.

Slow systems have a big impact on development and test productivity. Tests run longer or get skipped. Unnecessary summary tables and extracts take effort that could be used on new features and make batch run times longer. Highly skilled people twiddle their thumbs and surf the web waiting for jobs to finish.

BI systems have a performance problem when system speed impacts productivity. Here are some rules of thumb for understanding when your slow system is slowing down your team:

Batch and ETL Performance Problems:

• Developers check their email or surf the web while waiting for query results.
• Batch runs during business hours.
• Developers work in production because development or test environments are too slow.

Front end performance:

• You limit who can do as-hoc or analytic reporting because of performance concerns
• Reports take more than 5 seconds to open
• Filter/drill takes longer than 2 seconds
• No one outside the BI team and full-time analysts uses ad hoc
• Write new reports routinely require new summary tables or cubes

Getting to good performance

Every environment has its own challenges, but here are some common fixes. When the time comes to invest in system performance, we suggest splitting investments into quick wins (under $10,000 and less than two weeks), and larger efforts, which can take scale up to months and hundreds of thousands of dollars.

BI Performance Quick wins:

• Increase database server RAM. Database server RAM speeds up nearly all queries. If your cache hit rate dips below 98%, this is a likely source of improvement.
• Increase network bandwidth and add adapters. Network bandwidth between database, ETL, and reporting servers is a frequent bottleneck, and cheap to add. If your peak actual used bandwidth between servers is less than 70 MB/s, you have an opportunity to gain performance.
• Tune the 10 most resource intensive SQL statements. Tuning can be labor intensive, but the biggest wins usually occur in the top few statements. Consider only statements that run longer than 10 minutes for tuning.

Big wins:

• Move from a general purpose database to an analytic database. Columnar, cube, and appliance databases generate dramatic improvements over general purpose databases. Typical results are 10-100 times improvements, but the cost in rewritten ETL and front end is high.
• Switch from full refresh to change-only refreshes. Changing ETL to loading changes enables big performance gains, and lays the basis for more frequent refreshes and near-real-time.
• Add Flash storage. Flash storage is affordable and integrates well within SAN environments. Storage access performance gains can be in the 10x range.

System performance is not just an annoyance. High performing systems help your BI team deliver higher productivity and business value.

by Tom Victory, Principal Consultant

© DecisionPath Consulting, 2012

In my previous post, “The Analytics Blind Spot for Business Users“, I examined business users' perception of their BI capability, and how frequently they're not getting the value out of their business data.  This week, as part of our ongoing series on How BI is Being Used in 2012, I want to take a closer look at how well organizations are able to handle the three forces most responsible for the challenge of big data: volume, velocity and variety.

In our research, we found that most companies, regardless of size, felt that they were able to handle the volume (i.e. the amount of data being collected and analyzed) and velocity (i.e. the speed at which the data is being collected and requested).  Overall, 77% felt their IT infrastructure was adequate to handle the volume of data they would need in the near term, and 69% felt they had adequate infrastructure to handle the their companies' data needs when it came to velocity.

An area where they weren't as confident, was in the variety of data (i.e. the increasing sources of data created by new technology such as web logs, social media, RFID information, etc...); only 39% indicating that they had adequate infrastructure.

So what does this mean? Here are a few takeaways.

1. You might not be as prepared as you think for big (or even not-so-big) data

Business users are getting savvier. Although most companies' IT groups felt confident in their ability to handle the volume and velocity of data that their business users require, they may not have factored in the growth in demand from those users. It won't just be a matter of churning out more reports as more users become aware of the benefits of analytics (although we often see this "growing pain" for companies getting more mature with BI), it will also be a matter of providing more sophisticated analytical products.

2. Success with big data volume is tied to the success of enterprise data management

Many IT managers and executives view the volume of their data being a function of the number of transactions they're expected to collect in their transactional systems. It's actually quite a bit more complex; volume is also dictated by how a company is able to aggregate, consolidate and correlated that data.  In telecommunications and social networking, there's a theory called "Metcalfe's law" which says, essentially, that the value of a network grows exponentially as the number of nodes in that network grows linearly.  Applying this principle to big data, the connections between transactional data causes the volume of data to grow exponentially.

Managing this data would be challenging enough if there were a single version of the data, but for many companies, this isn't the case. Data often proliferates around the organization and each department creates their own data sources with their own definitions, leading to conflicting, inconsistent and sometimes just wrong data. Part of a good "big data" strategy, then, is good enterprise data management. EDM helps you get control of your data. It takes a holistic view of how data is managed across the enterprise and across its lifecycle, from initial creation through eventual retirement.

3. There may be better problems to solve with your data than variety

The infrastructure needed to analyze these new forms of data does not integrate with traditional transactional systems well. The investment needed to create an integrated view of this variety of data (i.e. structured and unstructured) is, for the near term, cost prohibitive for all but a handful of companies.

This is especially true for those companies who are underutilizing their current data. And as we've seen from the previous survey questions - where we see companies relying mostly on traditional reporting or scorecards and dashboards - there is a significant amount of value to be gained from improving advanced and predictive analytics. In short, companies looking to achieve near term ROI from BI would be better served by improving their basic BI capabilities.

Summary

Companies that are able to develop and manage the technical needs for big data - including the creation of more sophisticated analytics products, and the development of enterprise data management programs - are one step closer to realizing significant value from their analytics programs. But they're not there yet. The infrastructure is there, but the processes and skills needed to leverage that infrastructure for bottom line impact, more often than not, aren't.  Next week, we'll take a look at the non-technical challenges companies face with BI, and provide some insight in how to overcome them.

by Adrian Alleyne, Director Market Research
© DecisionPath Consulting, 2012

In my last post "Analytics In - Business Intelligence Out?" I discussed one of the key findings in our new research brief How Business Intelligence and Analytics are Being Used in 2012. In our research, we examined the terms that most resonated with business users when discussing the process of analyzing business information to support better decision making and improve business results.  Surprisingly, "business intelligence" rated fairly low compared to the terms "analytics" and "reporting." As I noted previously, the fact that "reporting" rated so highly suggests that many businesses executives have just scratched the surface of what BI can do.

In Business Analytics, Business Users Don't Know What They Don't Have

This week, we'll dig into a complementary finding from our report that suggests that business users don't know what they don't have. We asked our business users in Sales & Marketing, Operations and Finance to identify the tasks that were most important to their role. This included tasks such as "understanding the competitive landscape" (Sales and Marketing), "translating company strategy into operational plans" (Operations), and "planning forecasting and budgeting" (Finance).

We also asked our business users to rate how well their company's business intelligence and analytics capabilities helped them to accomplish these tasks.  Our respondents indicated that they had adequate BI for only 87% of their most important tasks. Given that fewer than half of business users indicated that they were using the more sophisticated styles of business intelligence (e.g., advanced analytics and predictive analytics), it would seem as though business users may feel as though they have adequate BI, yet are leaving much of the potential to leverage BI to improve their most important business tasks untapped.

Who's to Blame for Inadequate Business Analytics?

From our experience working with clients in business analytics, we see this scenario all too often. And even in the cases where they do know what they don't have and would like to use more advanced analytics, we often hear business users say that they have basic information, but it's hard to get and IT is non-responsive, and thus they have stopped asking for even entry-level BI, let alone advanced BI.

So while business users may blame their IT departments for lack of analytics capabilities, those same IT departments blame business users for not valuing analytics and making it a lower priority than other business projects. So who's to blame? Both. Neither. From a practical standpoint, blame really doesn't matter. What's more important is the ability to work together to identify the areas of top business priorities, explore ways that business analytics can improve those areas, and then put in place the systems and tools that will allow business users to execute in those areas.

It's important for these teams to figure out how to work together soon because in the next few years, they'll find themselves deluged with new types of data coming at them more rapidly and at higher volumes. In other words, they need to get ready to leverage big data.

Next week, I'll examine how companies feel about their ability to handle the expected increases in volume, velocity and variety of data over the next few years.

If you want a copy of the full report, you can access it here.

By Adrian Alleyne, Director of Market Research
© Decisionpath Consulting, 2012

We recently conducted a survey of business users, IT executives and BI teams on their use of business data. We posed the following question to all business users: "Different terms are used to describe the analysis of business information to support better decision making and improve business results.  Which of these terms do you use? (Select all that apply)"

The results were interesting: 87% selected the term "analytics," 80% selected "reporting," and only 47% of respondents chose the term "business intelligence." Given that we generally define business intelligence as the process (and technology) of analyzing business information to support better decision making and improve business results, it's interesting to see that so few business users view the term the same way.

Business Analytics vs. Business Intelligence?

Noted business analytics writer, Timo Elliot, examined the issue in his blog post "Business Analytics vs. Business Intelligence“in which he states "everybody has an opinion [on the difference between the two terms], but nobody knows, and you shouldn't care."  In one sense he's right; the difference could be a matter of semantics.

But as I look at the question posed in our survey, two thoughts occurred to me.  First, while it's hard to know if the selection of "analytics" is due to the phrasing of the question it's interesting that the term analytics is starting to become synonymous with the processes usually associated with business intelligence.  Second, the term "business intelligence" ranking so low is important for business intelligence teams, as it suggests that there's a disconnect for business executives between the technology and its application. We've found, in companies that are highly successful with business intelligence, that BI teams are tightly integrated with their business counterparts in the development, implementation, and measurement of BI projects, and that those projects are tightly aligned with core business processes.

Reporting only Scratches the Surface of BI

What also struck me about the response to this question was the frequency with which "reporting" was selected. True, reporting is a style of BI, but it's not a particularly sophisticated style when compared to things like advanced analytics and predictive analytics. If respondents are looking at traditional "reporting" as being representative of the value of business intelligence, it's no wonder so many companies have failed to realize significant ROI on the BI investment.

Getting More from Business Intelligence

This year, business intelligence is the top priorities for CIOs, according to researchers at Gartner. Given the responses we've seen in our survey, though, one has to wonder if it's similarly a top priority for business users. In order for organizations to get more from their BI investments, it's crucial that business users and BI teams work to get on the same page in how the technology is implemented, used, and measured.

Next time, I'll examine the second significant finding from our report: business users don't know what they don't have.

If you want a copy of the full report, you can access it here.

By Adrian Alleyne, Director of Market Research

© Decisionpath Consulting, 2012

True story: last night, as I headed home, I was all set to write a blog post about our forthcoming research brief on the use of business data for analytics: essentially looking at how business people and IT people view the adoption and application of business analytics very differently. Once home, though, instead of staring at the warm glow of an empty blog page, I was staring at the three inches of water stuck in my bathtub. And my blog plans, literally, went down the drain.  Half an hour later, as I'm standing in the plumbing isle of a major home improvement store in galoshes and overalls, I had to chuckle to myself thinking: this must be how a BI Manager feels sometimes.

In Business Intelligence, the Faucet Gets All the Glory

In the press and among analysts, much of the attention on business intelligence gets paid to the faucet. In other words, how your company's business data gets analyzed and displayed is constantly changing in new and shiny ways as new versions of the tools designed to present this information are released (much in the same way that new designs for faucets and other plumping fixtures are constantly on display).  But what about the underlying pluming that's responsible for pushing that information to that front end?

Often the consumers of this data don't think about (or care about) the infrastructure needed to deliver this information.  Until it breaks.  I recall the following question being posed on an open business intelligence forum recently: "Your company’s Business Intelligence solution just went down…no one can access it. What do you do?"

While numerous really intelligent folks provided well reasoned action plans to resolve the issue, it was Wayne Eckerson's response that stuck with me "Get out a stopwatch and see how long it takes for your phone to ring. If people start calling within minutes, congratulations! You have built a mission-critical BI/DW environment that people depend on to do their work and perhaps even drive core business processes."

I suppose it stuck with a lot of the readers, as it was voted the top response. It resonated with people, I think, because for folks who are responsible for keeping the day to day operations of the BI program going are often invisible to the rest of the company.

Note to Business Folks: Get out Some Galoshes for BI Success

Maybe the plumber analogy isn't the exact fit for business intelligence teams, but having to slog through messy data while new requests keep piling up, I'm sure sometimes it feels as though that's case.  While business people may not be able to roll up their sleeves to fix a BI "plumbing" issue, they should play an important role in making sure that the plumbing keeps flowing smoothly.

First, business users need to partner with their BI teams in the development of the BI program. In the same way that installing the plumbing in a new home requires one to know how the plumbing will be used, and what connects with what, so too should BI teams have input from their business users for how their BI infrastructure will ultimately be used.

Second, as business users get an appreciation for the complexities of keeping this infrastructure running smoothly, they are able to take more ownership of what gets put into the system. In other words, having your business users invested in data governance and data quality isn't an easy task, but with a greater appreciation for the clogs to the system that can occur without that sense of ownership, one is more likely to have a BI solution that continues to provide value the organization.

Next Time, More on Business Analytics Research

In case you were wondering, it was an unusually complicated hair clog, and the home improvement store had a number of innovative and effective solutions to solve the problem (just like with business intelligence). They just weren't as shiny and cool as the stuff in the fixtures aisle (just like with business intelligence). Next week, though, I promise to get more in-depth into our latest research findings.