Recently in Advanced Analytics Category

As companies look to derive more value from their data, many are looking to add advanced analytics to their BI capabilities.  One challenge that organizations find with taking that next step is finding individuals with the skills necessary to build, run and interpret analytic models.  In the past, tools like SAS, SPSS and R have been limited to a smaller subset of users in most organizations.  However, there are a number of new software offerings that are aimed at providing complex analytic capabilities to the masses.  They promote the ability for users to perform assisted advanced statistical analysis, such as market basket analysis, fraud detection and risk modeling without users necessarily having to understand statistical methods, such as regression, Chi-squared testing, ANOVA and K-means clustering in order to create and run the models. 

The question is whether making the tools easier to use will drive a larger population of users and derive more value from the data.  The challenge is that it may lead to a greater misuse of statistics and less confidence in the results by the business.  It takes a deep understanding of statistical methods and experience to not only create proper models, but also to interpret the results.  As eloquently described in Wikipedia; "Misuse of statistics can produce subtle, but serious errors in description and interpretation -- subtle in the sense that even experienced professionals make such errors, and serious in the sense that they can lead to devastating decision errors. Even when statistical techniques are correctly applied, the results can be difficult to interpret for those lacking expertise."

I applaud the BI vendors for addressing this market since I believe that we need organizations to be more analytically driven.  However, I don't believe that it's simply a matter of making the tools easier to use.  My hope is that they add instructive features into their applications with regard to proper use of statistical methods, how to select the right method for the analysis and how to interpret the results.  I'd also believe it's necessary to provide and promote analytic training, either virtual, classroom or both, as part of their offering. The result is that we end up with a larger pool of trained analysts which is good for both the vendors and their customers.  In addition, organizations looking to achieve more value from their BI investment win as well.

Each year I form my opinion based on the observations from five sources; our customers, industry conferences, articles, social media and BI software vendors.  2010 proved to be an interesting year on many fronts.  My observations from 2010 are as follows:

1)  Larger numbers of existing BI programs are maturing beyond managed reports, ad-hoc queries, dashboards and OLAP than in years past.  Companies are increasingly looking to squeeze more value out of their BI programs via advanced visualization, predictive analytics, spatial analysis, operational BI and collaboration.  This is driving the rise of analytic databases (ADB's) and new analytic applications and features.

2)  The number of failed BI projects continues to remain high.  While the industry has learned, and documented, the reasons of why projects fail, it hasn't done much to stem the tide of failed projects.  From my perspective, and I'm sure most would concur, the overarching reason is because implementing successful BI projects is hard.  It requires a balance of strong business involvement, thorough data analysis, scalable system & data architectures, comprehensive program and data governance, high quality data, established standards & processes, excellent communication & project management and, experience.  I don't necessarily see things changing in 2011 unless companies:

• institute and enforce Enterprise data management practices

• ensure high levels of business involvement for BI projects

• institute measurable, value driven, metrics for each BI project

• realize that offshoring BI projects, except for simple staging ETL and simple BI reports, doesn't work well for BI.
  

3)  Much of the old is new again.  Master Data Management (MDM), dashboarding and predictive analytics are not new concepts, but they did see a strong reemergence in 2010.  One of the challenges with implementing integrated dashboards and predictive analytics was that they were often missing from the enterprise BI software suites.  It seems that the major BI vendors finally started listening to their customers are new capabilities started rolling out.  MDM was simply repackaged to include more than just the data warehouse and found traction in organizations struggling with defining their master data and keeping it consistent across their disparate systems.

4)  BI wants to be agile. We've always recognized the high cost and long lead times for implementing BI, but it seems that the customer has finally said 'enough' and BI teams are listening.  They are looking for new ways to implement BI and finding that many Agile practices, such as smaller, focused iterations, daily scrum meetings, prototyping and integrated testing help speed up the velocity of BI and augment the communication between the business and IT. In 2011, I expect to see BI practitioners better understand what Agile practices work with BI and which one's don't translate as well.

5)  There was increasing interest in BI in the Cloud, but fewer takers than expected.  Infrastructure as a service is still a bit too technical for many BI programs to implement and concerns over data security remain high.  There are also concerns over performance, both hardware and network.  BI software as a service products are continuing to mature but  the biggest hindrance for user adoption in 2010 appeared to be poor ETL capabilities and non-extensible identity & access management.    

I'm always interested in you reaction and views.  Please feel free to post your comments and take on 2010.  

I recently returned from the TDWI spring conference in Chicago, where I spent 3 days meeting attendees, sitting in on parts of some classes and also presented a course on BI in the Cloud.  I am always interesting assessing the trends in BI by listening to what the attendees have to say and gauging course interest.  While it's unscientific at best, it has been a fairly good predictor over the years of where the focus on BI has gone.

This year, the topics that seem to generate the most buzz were:

• Agile BI
• Data Visualization
• Statistical Analytics
  

I spoke with a number of attendees that had mature BI programs and were interested in new ways to add value.  They had conquered reporting, OLAP analysis, events & notifications, dashboards and scorecards and felt as though their programs weren't keeping up with the current needs of their users.  Their users were becoming more sophisticated and looking for new ways to analyze their data.  They were also looking for ways to increase the velocity of their project iterations as users have started to look to software-as-a-service (SaaS) to address data and functionality gaps. 

The attendees that I spoke with that are new to BI were concerned primarily with the well publicized rate of failed BI projects.  Some worked for companies that had past failures with BI and were interested in how Agile project management might help.  They were also interested in technologies that might facilitate Agile projects.

Only time will tell whether Agile BI, data visualization and advanced analytics will become this year's major trends in BI.  However, it's exciting to see strong interest in areas that have the potential to change the way we deliver BI and the value of what we deliver.  What's interesting is that these aren't new concepts in the world of project management and data analysis, it's just that we may have matured to a point where they become front and center in our focus.

I'm interested in your thoughts on what you think will be the major trends in BI in 2010.  You can leave your comments below or feel free to email me at sdine@datasourceconsulting.com.