We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Blog: William McKnight Subscribe to this blog's RSS feed!

William McKnight

Hello and welcome to my blog!

I will periodically be sharing my thoughts and observations on information management here in the blog. I am passionate about the effective creation, management and distribution of information for the benefit of company goals, and I'm thrilled to be a part of my clients' growth plans and connect what the industry provides to those goals. I have played many roles, but the perspective I come from is benefit to the end client. I hope the entries can be of some modest benefit to that goal. Please share your thoughts and input to the topics.

About the author >

William is the president of McKnight Consulting Group, a firm focused on delivering business value and solving business challenges utilizing proven, streamlined approaches in data warehousing, master data management and business intelligence, all with a focus on data quality and scalable architectures. William functions as strategist, information architect and program manager for complex, high-volume, full life-cycle implementations worldwide. William is a Southwest Entrepreneur of the Year finalist, a frequent best-practices judge, has authored hundreds of articles and white papers, and given hundreds of international keynotes and public seminars. His team's implementations from both IT and consultant positions have won Best Practices awards. He is a former IT Vice President of a Fortune company, a former software engineer, and holds an MBA. William is author of the book 90 Days to Success in Consulting. Contact William at wmcknight@mcknightcg.com.

Editor's Note: More articles and resources are available in William's BeyeNETWORK Expert Channel. Be sure to visit today!

I'm getting concerned about the data warehouse. It has served us well, but can the current profile of data warehouses out there handle the next 10 years or will widespread changes be necessary? Consider that most data warehouses out there are not best practices by definition and are therefore dumps of operational data where history collects and reports are run from. This only solves some of the challenges associated with going it alone with just operational data, which are:

Data access
Reporting capabilities
Concurrency between query and operational needs
Structure for data access
Data quality for data access
Data integration
Storage of history data

Notably, it is the concurrency and history issues that instigate many data warehouse programs. However, integration is largely limited to data sharing a common database instance - which is good, but leaves too much complexity to the data access layer, where the end users find the data access tools too complex already. Building summaries and making sense of the data warehouse structure and data, especially without metadata, which most DW lack adequate levels of, is exasperating so current users mostly skim the surface of their true needs.

Also, data quality is only addressed in data warehouse programs out there selectively. Many remain afraid to change operational data, even if it is wrong. It needs to be fixed operationally anyway, and that just isn't happening enough.

So, how is data warehousing supposed to fit into this new world of data explosion, real-time requirements and a need for process-orientation?

1. We can't continue to delay the calculation, assimilation and distribution of master data until the data warehouse
2. Business intelligence, as a discipline, must be extended beyond reporting and even dashboarding and get involved in business processes using enterprise information integration and operational business intelligence approaches; these open up the possibilities beyond post-operational, after-the-fact BI
3. We need to embed business intelligence in operational processes and try a lot harder to fix data quality in the operational environment; the longer action is delayed, the less valuable it is; this can be the equivalent value of thousands of end-user data access licenses

This world requires integration between business units. It requires the understanding that information is a most-important business asset.

Of course, we could improve our data warehouses too with data quality, metadata, deriving data and true integration. In reality, for most, this is needed as well as a change in direction that focuses on the augmentation of the data warehouse with these new concepts. Most data warehouse programs will see these changes come one way or another in the next few years.

Technorati tags: data warehouse, information management, enterprise informatoin integration

Posted March 16, 2007 1:00 PM
Permalink | 2 Comments |


William, finally someone with a data warehousing pedigree dares to raise these issues. Thank you.

The next wave of BI (and what is the point of discussing data warehousing without BI) will be proactive, real-time, operational and integrated with business processes that extend beyond the firewall. This going to drive the DW to some radical new functionality such as: watching minute details, not just aggreates, and alerting when something pops out of bounds; reorganizing in an instant, and managing multiple org’l schemes simultaneously; embedding analytical processes in operational processes, making your whole organization smarter and freeing knowledgeable people to do more valuable work.

Almost all BI products that are used with DW's are read-only. This exposes the reporting prejudice of DW methodologies. But people don’t sit in front of their tubes waiting to be bathed with integrated, lightly aggregated, time variant data. They have ideas of their own and they create data too. They share it. This is beyond the capability of DWg.

And then there is the Single Version of the Truth. This is the most misguided idea to come out of DWg. How can a set of modified secondhand data be the SVT? What power on earth or the heavens can be imbued in a data modeler to come up with the “truth” in something as complex as an organization conceived and mangled by human beings? SVT assumed that even external data is part of this scheme. That is impossible. When you cross boundaries, you have to be a little more expansive in your thinking.

We do need metadata, I agree. But what kind of metadata? I'm holding out for semantic models because they replace simplistic ideas like SVT with context, partial truth, complex relationships and a way to traverse these graphs efficiently.

As usual, I agree with William, but we need to be clear about HOW we're going to move to the next level.

Will existing data warehouses (built using the technologies and best practices of the late 90s and early 2000s) be capable of sustaining business value over the next decade? No way. The problem? The rate of change has increased, and the world's existing data warehouses were not built for change.

Anyone who can predict what is in store for the next five years will be incredibly prosperous. However for everyone who isn’t clairvoyant, we need to build for change and anticipate that many of our business expectations and predictions will be wrong! Build your data warehouse to easily adapt to change, and you don't need to know what will happen in the next 5 years.

Leave a comment


Search this blog
Categories ›
Archives ›
Recent Entries ›