Blog: Dan E. Linstedt

« Automorphic Data Models | Main | Part 2: Secrets of the Masters »

Secrets of the Masters - Part 1

It's been a while since my last blog entry, my apologies. I've been heads down building companies lately, and they seem to be starting to gain steam. All that aside however, I've been thinking quite a lot over the years about these notions we hear about: top 10 this, top 10 that... most of these are about "mistakes" we can make, followed by short sound bytes about what or how to look for answers. There are books and books filled with great information about building, maintaining and deploying an Enterprise Data Warehouse, and then there are architectural discussions from John Zachman, Bill Inmon, and others. All good information....

But what I've been thinking about is a practical guide, a good sense guide to how to get from point A to point B, and then do it over again, repeatably, reliably, and consistently. Not just to do it the first time, but also for those of you getting in to 2nd Generation Warehousing, or DW2.0, what does it mean? How should it be approached?

Please keep in mind, I'm no expert, and I hope I never claim to be, that title is reserved for people like Colin White, Bill Inmon, Claudia Imhoff, Ralph Kimball, and a few others who really are the true thought leaders. No, what I want to share is a culmination of 15+ years in the industry of building enterprise data warehouses and business intelligence systems. What to do, what to look for, how to do it, how to make it work.

I'm not (hopefully) going to dive in to the nitty gritty of it all (I've done that in postings in the past), but I'm going to stick to the architectural roots, and try to make business sense out of all of this.

For starters, we (most of us anyhow) are familiar with Dr Kimball’s approach to enterprise data warehousing; and when we reference his books in the LifeCycle Toolkit, we talk about things like Star Schemas, Enterprise Service Busses, and Staging Areas. But let me back up...

There are a few things that the CIO should be discussing and contemplating when building ANY enterprise based systems:
1. Singularity - cost of management, maintenance, organization, and flow - everything should have a center or master point of contact. This includes data, people, process, organization, and so on. Business have singularity (usually) in their organizational structures/reporting structures, why should data, and business process be any different?
2. Repeatability, Standardization - Once a success has been established, it should be repeated again, for a lower cost, higher value, and measured/monitored effort. CIO's need to focus on things like ITIL, CoBIT, CMMI, PMP, Six Sigma, and if a CIO doesn't understand these terms, or can't bring them to bear within their own organization, they should either be sent to school to learn these tools, or be let go from their position. The effect of deploying these measures within a corporation is to enable the corporation to grow, safely and collectively with practices that scale. The business gets better and better at singularity and success. Small shops should focus on leveraging these ideas at a high level, getting the corporate culture used to operating this way.
3. Measurable - Without standardization of people, processes, and data, measuring and estimating the next big project could turn out to be a disaster. Measuring the process in the Estimation cycle, or in the projection to completion cycle, or in the current effort cycle (level of effort aka LOE) is extremely important. Understanding the measurements of the people AND the data AND the processes involved go a long way to controlling project cost overruns, and budgetary concerns. They also help to point out where things go wrong, have gone wrong or are in danger of going wrong in the near future, thus enabling the business to navigate around the issue or solve the issue, which ever they choose to do. If you can't measure the effect of the people, the systems, processes or the data, then you won't be able to improve what you are doing, and thus make a repeatable success next time.
4. Automatable - once the CIO has gotten a handle on the processes, and the corporate culture has turned the corner (gotten used to standardized practices, and version control, document control, PMP and six sigma practices), then the CIO should look at automating parts of the business.

Why am I rehashing old ideas here that we have known for I.T. in general for years? Well, guess what: the EDW teams over the past 10 years (for the most part) have claimed: no standards to follow, it's a different project, every single customer is different, all pieces are special, new, and are exceptions to the rule... You name it, the EDW projects have done it (in a blaze of glory no doubt) in order to deliver on time.

Ok, so in walk methodologies that the Big 6 (now final four if you can call them that) from the various consulting companies. I'm guilty of that too - I wrote one of these behemoths in my past life... They are massive in documentation, enforce rigid standards, and usually are waived in the clients face as proof that the company knows what they are doing.

This isn't the problem, no the problem isn't that these methodologies exist, nor is it a problem that they are followed (which in most cases they are...) The problem is: when the consulting companies leave your realm... what do they leave you, the customer with?

Empty arms, lack of knowledge, minimal training... They turn over the project, walk in to do their song and dance, show what they've provided - but no one on your team understands the infrastructure, the standards, the closely guarded methodologies they used to build the huge solution set. It's like walking in to the empire state building in NY and not knowing where the elevators are because you can't see them; yet you know they are there.

CIO's with all the off-shoring, in-shoring, on-shoring, near-shoring, and beach belly wagging (just kidding)... have gotten lost in the fray here. Companies need to get back to doing business the way they do business, and they need to get back to contracting consulting companies that do business they way THEY need it done. Which means internal controls, internal standards, internal documentation, internal improvement processes...

But I digress. Back to the point. Enterprise Data Warehouses need guidelines, standards, processes, architectures, and workable solutions - otherwise you'll be stuck in a quagmire of mud for a bit, while stretching to reach the DW2.0 bank of solid ground.

These entries will start at high levels, and dive down a little bit to discuss how, and what we do to build successful systems. Remember, these entries come from my own personal experiences of working in the trenches - I'm open to comments on all forthcoming entries to see what other knowledge we can share.

Cheers,
Dan L

Share: