Blog: Krish Krishnan

« Which Appliance fits your needs? | Main | Is the SMP architecture suited for the Data Warehouse Ecosystem »

Historical Data - Costly Maintenance

Historical data (greater than 3 years from today), Is it needed all the time by your users? can you afford to move your historical data to an offsite location, will you be able to get it back and have it available in the data warehouse when required? if the answer is yes, then you are in good shape, if the answer is no to any of these thoughts, then read on.

Look at the overall cost of what historical data does to your bottom line

1 Increases the cost of storage
2. Increases the response time of your queries
3. Increases the time you take to load data to the warehouse

Well most of us know the obvious issues as listed above, but there are other issues that are often overlooked

1. Metadata and Master Data for the historical data needs to be maintained in the data warehouse.
2. If ETL code was developed in increments over the historical data, it is an additonal overhead that needs to be maintained.
3. If the historical data is archived and needs to be brought back then there is an issue of missing the metadata and thus losing the interface to the data.
4. Integration of the historical data to the new data always causes issues.
5. Data content between historical and current data fields might have changed, this is an impact when you want to do comparison reporting.

When you start looking at the overall impact of both maintaining and not maintaining the historical data, you will be confused as to which way to proceed. A correct decision to be made in this regard will require you to look at the value that this data will provide to your business. In other words, before you decide how to manage a data lifecycle within the data warehouse, involve the business, by taking this problem to your data governance and steering committee. Start showing the business the pains that they are facing from a volume of data and explain to them the different options of how you can mitigate this pain. One such option is to consider a new platform to store your historical data onsite at alower cost of ownership.

A data warehouse appliance will help you achieve the ability to store your historical data and still not pay in storage and reloading costs. All of the appliance vendors are ANSI SQL compatible and you should have no issues in pointing your BI tools to this platform for analyzing historical data. If you need to do comparison analysis, you can still build a special dataset on this platform and then bring it to your data warehouse for further reporting.

Bottom line is that there are new and flexible options in the technology for your data warehouse solution architetcure, how soon will you adopt to it depends on how intense your pain and financial drain is.

Share: