Blog: Krish Krishnan

« Why an Analytical Data Warehouse Appliance | Main | Another new for Microsoft - CRM at Grocery Supermarket »

Number Crunching Cost Effectively

Hello and Happy New Year 2008.

In my last blog, I started out on discussing why we need an Analytical Data Warehouse and mentioned about Columnar Databases. Continuing on that thought, whenever we talk about Predictive Analytics and Data Mining exercises, we all think of the PhD analysts and the complex algorithms they write, and the massive volumes of data they ask for before they come back with thier models and analysis. Bottom line, when looking at these exercises even today they are expensive and complicated. We cannot replace the statisticians since they are the SME's, but we certainly can look at providing them better infrastructure and accelarate the analysis time.

Statistical models or Analytical models as we all know it, take a large multidimensional dataset and analyse the same through complex calculations to arrive at results. A traditional database engine can accomodate this requirement, but we run into the same old OLTP bottlenecks on shared everything database execution causing CPU contention, heavy IO and slower network bandwidth.

While we get results, they are not quickly available as desired. If the results are needed quickly then the infrastructure costs go through the roof. Due to these catch-22 issues many organizations often end up dismantling their Analytical and Statistical BI initiatives.

This is where once again I see a future for the columnar database. due to this technology, we can compress the huge multideimensional data and leverage all the benefitsthat is available from this database technology. On this note I also point out that Data Warehouse Appliances can provide an alternate platform to implement this database.

By implementing the analytical database on a columnar database technology, the application server for the analytical database can offload all the heavy and complex calculation to the database engine.

While we all may be skeptical about the whole concept, I can visualize the merits of this solution and the underlying blueprint that it can provide from a platform perspective.

From the next blog series that I will be contributing, we will continue to explore industry specific issues and share the situations faced in the field everyday in different disciplines of BI.

Share: