Blog: Krish Krishnan

« A Scalable Architecture | Main | Which Appliance fits your needs? »

The Need for Speed

A very simple word in the English language that keeps data warehouse architects, database administrators, data warehouse project managers up at night is Speed. Business users demand more speed in terms of performance and scalability (increased data volumes) constantly. A consequence to this is a direct increase in the cost of the data warehouse, whether you add services to augment staff or add hardware and storage. In the end of this cycle, there are users who are left with solutions that perform below their expectations and IT is left with a big gaping budget deficit. How do we win this constant battle, here is where the data warehouse appliance comes in handy.

Although the technology and the architecture itself is not new, if you consider Teradata as an appliance, the rest of the vendor offerings are new and have been introduced in the last five to seven years. The hesitation amongst the IT user community to adopt to the appliance and include it in the technology stack is based on the fact that it is new, and that appliances are built on database platforms which are community developed.

What is it that makes the Appliance work better than traditional solutions when we talk about scalability and performance? it is the architecture of the appliance that makes the difference. If the argument is that one can make any database platform scale and perform, the answer is yes, being a DBA myself, I can see how it can be done. But when one starts engaging in that exercise there is an associated cost that is not cheap and often the end solution does not satisfy the user needs to the fullest extent.

By introducing the appliance into the data warehouse architecture, we are not claiming to reduce complexity in your architecture. But what we can achieve by augmenting the appliance to the data warehouse are tasks like table load balancing for data volumes, partition pruning etc can be simply achieved by utilizing native commands that are provided in the appliance which in a traditional solution would need extensive planning and coordination between database administrators, storage administrators and end users and will require downtime from an end user perspective (thought rdbms technologies can support online table and partition management). If the question is about the massively parallel processing (MPP) technology that provides the much needed performance boost in respect to maintenance and support, the MPP engine is developed and supported by each appliance vendor like any other technology. With respect to the appliance management itself, interfaces provided by the vendors to manage the appliance like how you manage databases. With some of these tangible benefits, a short term pain to adopt and augment a new technology will potentially free up the data warehouse IT team to focus on the needs of the core solution in the longer term.

The goal to consider the appliance in the data warehouse architecture should be to provide end users with a performing and scalable solution, and controlling the cost of the data warehouse. The idea is to add the appliance and not replace your existing architecture, unless the existing architecture needs a complete replacement. BI tool vendors are moving to adopt the appliance as the platform to offer low cost packaged solutions. Other tool vendors in the data warehouse space will follow suit in the future.

The appliance will reduce the workload on the data warehouse when incorporated into the solution. It is a lower cost to execute large queries on the appliance than the data warehouse itself, simply because you allow the data warehouse to be available for other users. If you start examining the remaining areas and it s benefits, this technology is here to stay.

To summarize, when you quantify the success of your data warehouse project, the business goals that are your requirements for the data warehouse should be met with information delivery that adds increased value to enable the business to make informed decisions faster. The appliance is an enabler to that goal and definitely not a detriment. The budget aspect and other ideas will be blogged in the days to follow.

Share: