We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Blog: William McKnight Subscribe to this blog's RSS feed!

William McKnight

Hello and welcome to my blog!

I will periodically be sharing my thoughts and observations on information management here in the blog. I am passionate about the effective creation, management and distribution of information for the benefit of company goals, and I'm thrilled to be a part of my clients' growth plans and connect what the industry provides to those goals. I have played many roles, but the perspective I come from is benefit to the end client. I hope the entries can be of some modest benefit to that goal. Please share your thoughts and input to the topics.

About the author >

William is the president of McKnight Consulting Group, a firm focused on delivering business value and solving business challenges utilizing proven, streamlined approaches in data warehousing, master data management and business intelligence, all with a focus on data quality and scalable architectures. William functions as strategist, information architect and program manager for complex, high-volume, full life-cycle implementations worldwide. William is a Southwest Entrepreneur of the Year finalist, a frequent best-practices judge, has authored hundreds of articles and white papers, and given hundreds of international keynotes and public seminars. His team's implementations from both IT and consultant positions have won Best Practices awards. He is a former IT Vice President of a Fortune company, a former software engineer, and holds an MBA. William is author of the book 90 Days to Success in Consulting. Contact William at wmcknight@mcknightcg.com.

Editor's Note: More articles and resources are available in William's BeyeNETWORK Expert Channel. Be sure to visit today!

Tanjian Norman commented on a post from 2005 as follows:

"DW is dead? It seems that EAI/EII or more importantly service enablers like SOA are complementary to a DW environment. Please elaborate on why you think DW is positioned for replacement by EAI/EII technologies."

He was responding to my post in which I said, among other things: "DW will eventually go the way of EAI. The extra data store in the picture is redundant and the market eventually drives out inefficiencies."

I thought I would bring my response into a new post...

No, DW is not dead. It's alive and well and thriving. Almost every midsize and up company has at least a semblance of one. A robust data warehouse is highly desired and sought after everywhere. My point is about the future - when exactly I do not know. I agree that EII is complimentary-only to data warehousing today, but its merits should surely be considered.

Data warehousing is evolving. New data warehouses and data warehouse rearchitectures are well advised these days to consider not building a pure batch-loaded data warehouse where all analytic calculations, including master data and all reporting is done. There are several layers of calculations, functions and even data that are no longer necessarily part of a robust data warehouse reference architecture.

1. Master data calculations - Master data is not ideally calculated in a downstream data warehouse. It is needed in the operational environment as well as the data warehouse. As time goes on, the data warehouse will be a receiving system for the master data.

2. Operational business intelligence - I blogged about this here. There is certainly a lot of calculations that do not, or cannot, interact with the data warehouse in order to be effective. This can go well beyond basic operational reporting from a single system.

3. Yes, EII. EII is able to facilitate multi-system operational reporting and business intelligence. Some clients believe that several of their data sources do not need to be fed into the data warehouse if they can run EII queries and eliminate the redundancy of having the data in multiple places. EII can handle multiple databases in multiple formats, referential integrity, XML and basic transformations. EII still has a long way to go (query tuning, 2 phase commit, business metadata, memory constrained, etc.) and data warehouses are still absolutely vital, but it shows promise and is another factor chipping away at the data warehouse requirement.

4. Modern ERP - There was a time when ERP vendors debated that the data warehouse was not necessary. When this was proven untrue, they provided packaged data warehouses so at least they kept some of that business too. In the background, they've continued to add functionality to the base ERP to keep chipping away at the data warehouse requirements. Today, one of the striking things about ERP systems is they are keeping history data indefinitely. Having a historical data repository used to be one of the main reasons for building a data warehouse, but that is not always a data warehouse requirement any more.

And finally, a "real-time" data warehouse is evolving to look more and more like an ERP system itself, with real-time feeds of operational data, triggers and analytical applications. So, the definition of data warehousing may be changing to keep the term active, but the data warehouses themselves are evolving.

I hope this helps. Feedback welcome, which could be interesting...

Posted August 14, 2006 9:14 AM
Permalink | 2 Comments |


Hi William

You make some interesting points in this post (and others within the blog). In particular I like the way that you are looking back at the evolution of data warehousing and projecting where it’s going, given business pressures, technology developments (including SOA and real time business intelligence technologies) and the changing expectations that people have.

This echoes may of the themes in my ebook “In Search of Insight” which can be downloaded free from http://www.seewhy.com.

This extends your discussion of both real time information and operational BI, both of which are significant trends for the industry.

In particular, we’re finding that these two are often connected. Why do you need real time information? Probably because there is a pressing operational problem that needs fixing. This of course leads directly to process, and an area that the DW struggles with. The data warehouse is a data centred store, not a process centred store. So wherever you need intra day information (typically problems or opportunities) presented in the context of a process state, the data warehouse is the wrong platform to achieve this.

The conclusion which quite a few people are coming to (analysts included) is not that the data warehouse is dead, merely that Business Intelligence will be done (and is already being done) much more broadly than solely centred around the data warehouse. Operational / process centric BI needs to have a very low latency store, and atomic level detailed data including process state. Latency is critical simply because you cannot automate a corrective action without real time data. Reporting that the process broke yesterday simply doesn’t cut it.

To address operational BI needs of this type, a different kind of architecture is required than a traditional data warehouse. As a result we see the emergence of a new category of BI technologies using Event Stream processing which are designed for this task.

These trends and technologies don’t mean that the data warehouse is dead, rather that its role is changing from ‘the place where you analyze data’ to ‘the system of record.’

I believe data warehousing is here to stay. It is the only place where you can keep track of your history and changes that has happened. Nothing in EAI/EII/ERP will ever replace that functionality.

I further believe that we are on the verge of building data warehouses that will outlast several ERP changes. Why? Well, for many it still comes as a nasty surprise that you will have to rebuild your warehouse every time you change one of the surrounding systems. Only in the last few years have we found methods to model our warehouses with the flexibility necessary to cope with changes like these.

If you are interested in the latest developments in data warehouse modeling, please look at our homepage: http://www.intellibis.se and read up on anchor modeling.

Leave a comment


Search this blog
Categories ›
Archives ›
Recent Entries ›