Blog: Krish Krishnan

« July 2008 | Main | September 2008 »

August 26, 2008

An interesting move....

This week has seen a significant announcement from Aster Data Systems, an upcoming data warehouse appliance (implemented as a software appliance on commodity hardware, website www.asterdata.com). Aster announced the implementation of the famed Google MapReduce framework into is architecture.

There are fellow analysts and other pioneers in the database field, who question the validity and the scalability provided by MapReduce. My goal is not to debate on MapReduce, but what I'm fascinated by and believe will help users especially in a large scale data warehouse, is the ability to provide a language independent search capability.

This feature can be implemented by FAST or ENDECA, but what the framework from Aster will help bring is the scalability and all the bells and whistles of a data warehouse appliance, with the search capability of a Google like architecture.

Yes, we are a long ways away from achieving a true Google like search on the data warehouse, but the new crop of vendors are taking baby steps to get to that point.

Greenplum has also announced that in a future release of their software this year they are implementing the MapReduce framework.

We will see more of these trends coming from the data warehouse appliance and new business intelligence vendors. The traditional RDBMS platform may think this direction too, but they definitely have a long pace to catch-up. Though all this is bleeding edge, these are the features that will dictate the future of the data warehouse and business intelligence.

Share:

August 22, 2008

The importance of Infrastructure teams

In my opinion the most overlooked team in any project is the infrastructure team. This is very true of data warehouse initiatives. Most often the infrastructure team is taken for granted as the "in guys" who know how to put together machines, storage and databases. This kind of an assumed role and responsibility often leads to missed opportunities to communicate clearly and then there is a lot of rework or redesign etc.

We need to move away from this attitude to start involving the infrastructure team in any data warehousing project from the get go. This will enable the infrastructure team to start understanding the system requirements and expectations at a much better and deeper level, which will help the overall solution design and development.

Involving the DBA team for example from the last stages of requirements finalization to logical data model design, will enable them to design a physical database design with the right kind of key and data structures that will require minimal tuning or any other rework once designed and implemented.

Infrastructure is a monumental piece of effort in any data warehouse project and with the rapidly changing technology advancements, the sooner the better.

Share:

August 17, 2008

All said and done

Whatever may be your technology and architecture, there are the layers of infrastructure that continue to worry us about the performance of your data warehouse and business intelligence solutions.

Broadly classifying the infrastructure, you have the hardware, network and storage. Though each individual layer has had commendable improvements, the holistic solution is still weak. How can we achieve the optimization across the network layers?

Here is where companies like Aster Data are making inroads. The software as an appliance with customization across different layers of infrastructure and in particular the network has enabled this appliance to achieve some "cool" optimization. Ofcourse there are multiple ways to solve this optimization problem, the network optimization is one such layer and yes there are other solutions in the offering.

Fundamentally until we all start thinking about optimization from the data design to infrastructure design, we will not achieve any throughput, whatever the case may be.

I agree that there are several innovations current and in stealth to address the issues, but wait a minute, do we have anything that is end to end i.e. data to infrastructure?

Share:

August 8, 2008

Is there a "business value" for the ODS?

A question that has been asked by many of the data warehouse community is, with the advent of near real time data integration into the data warehouse what is the "business value" for building an ODS. I would like to see your comments on this topic. A new article on the same subject us underway and will be published in the next month.

Share: