I'm getting concerned about the data warehouse. It has served us well, but can the current profile of data warehouses out there handle the next 10 years or will widespread changes be necessary? Consider that most data warehouses out there are not best practices by definition and are therefore dumps of operational data where history collects and reports are run from. This only solves some of the challenges associated with going it alone with just operational data, which are:
Concurrency between query and operational needs
Structure for data access
Data quality for data access
Storage of history data
Notably, it is the concurrency and history issues that instigate many data warehouse programs. However, integration is largely limited to data sharing a common database instance - which is good, but leaves too much complexity to the data access layer, where the end users find the data access tools too complex already. Building summaries and making sense of the data warehouse structure and data, especially without metadata, which most DW lack adequate levels of, is exasperating so current users mostly skim the surface of their true needs.
Also, data quality is only addressed in data warehouse programs out there selectively. Many remain afraid to change operational data, even if it is wrong. It needs to be fixed operationally anyway, and that just isn't happening enough.
So, how is data warehousing supposed to fit into this new world of data explosion, real-time requirements and a need for process-orientation?
1. We can't continue to delay the calculation, assimilation and distribution of master data until the data warehouse
2. Business intelligence, as a discipline, must be extended beyond reporting and even dashboarding and get involved in business processes using enterprise information integration and operational business intelligence approaches; these open up the possibilities beyond post-operational, after-the-fact BI
3. We need to embed business intelligence in operational processes and try a lot harder to fix data quality in the operational environment; the longer action is delayed, the less valuable it is; this can be the equivalent value of thousands of end-user data access licenses
This world requires integration between business units. It requires the understanding that information is a most-important business asset.
Of course, we could improve our data warehouses too with data quality, metadata, deriving data and true integration. In reality, for most, this is needed as well as a change in direction that focuses on the augmentation of the data warehouse with these new concepts. Most data warehouse programs will see these changes come one way or another in the next few years.
Posted March 16, 2007 1:00 PM
Permalink | 2 Comments |