Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt,

About the author >

Cofounder of Genesee Academy, RapidACE, and, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on

Here is another installment of the secrets of the masters. Quite frequently customers and IT alike complain about how difficult it is to gather business requirements. They discuss the pain of having to "get together" for a day, or for a week-long process to write down and document business processes, and ultimately their needs and desires for a new BI/EDW system. Any good analyst worth their salt has battle-scars from negotiating these treacherous grounds.

We've all walked in to an environment with a blank white-board and asked: business, please give me your requirements, only to be confronted with: "What can you provide to us?"

I'm here to describe a different way to you. This is an ancient technique and requires a flack-jacket to be worn by the IT participants at all times. Remember: nothing is personal, this is only business. No really, I'm not a Zen master, but a Zen master once said: it is far easier to tear something down than it is to build something up. So with that in mind, here we go... (I'm kidding about the ancient part...)

There are many of you in the BI and data integration world who handle BI requirements in a similar fashion: go through the months and months of drudgery discovering business processes, and "requests" for new systems design. Then go through all kinds of long lasting meetings to pull together a buy-off, and write up a business requirements and a technical requirements document. Then throughout the design and build phase of the project, have the users "slip in new requirements and larger scope because they forgot to tell you something."

But this process is incredibly painful, requires a lot of money up front before business users can "see" anything, and requires diligence on behalf of the IT and business folks to see it through. Now don't get me wrong, I'm NOT saying this is a bad thing, I am saying that it simply takes too long and there is a better way up Mount Everest.

Ok - enough with the bad jokes already... How do I do this?
For one, you need to shift your thinking about integration of data sets into your data warehouse. I've blogged on this before. There is a paradigm shift in the works for auditability and compliance, and it basically says: move your business rules DOWNSTREAM of the EDW. That's right, take a deep breath and swallow. Placing business rules upstream of the EDW will lead you back to the old techniques of waiting and waiting for business requirements. Moving the business rules DOWNSTREAM and implementing them coming out of the EDW allows us to see new light on gathering business requirements.

Easy for you to say... I still don't quite get it....
Right. Have you ever walked in to a room full of business users and started describing the data that their systems are "collecting today" from an integrated stand point? If you haven't, you should try it. If you have, then you know: Business users at that point are quick to the draw to point out why you're wrong, where the systems are wrong, what the problems are with the systems you are talking about, and of course - why they have their own special Excel spreadsheet that they built to FIX this problem.

The point is, you can literally start fires with your pen you are writing so fast... These are the missing business requirements that you've been seeking so long and hard for. It requires a flak jacket because you cannot take it personally.

By moving the business rules downstream of the EDW, we can load RAW and AUDITABLE data "as-it-stands" into the EDW. From there we can produce something called an "AS-IS STAR SCHEMA." The AS-IS Star shows raw level grain, with un-doctored and uncensored, and un aggregated data sets. You can then share with the business users "this is the way your source systems (once integrated) are currently capturing data, and by the way, these are the results of your source systems executing your business processes."

They very quickly are more than happy to tear it down, shoot holes in it, tell you why it's wrong and why it won't work. Again, if you're willing, you can gather nearly all the requirements you need for phase 1 within a day, or 2 day session. This reduces the cycle time to delivery of your EDW environment, increases the visibility into all the "work-arounds" the business users are currently engaged in to "get the source systems to do what I want."

I've been using this technique for 16+ years, and it hasn't failed me yet. But again, it requires a strong will, and to move the business rules downstream. AFTER you've collected the business requirements, then you can build integration processes to take the data from the AS-IS stars into the "business release" star schemas. Also, by moving the business rules downstream you can meet accountability, auditability and compliance in your EDW.

This is one of the most powerful secrets of the masters available within Integration projects, whether you are executing SOA, ESB, Web Services, or EDW / BI projects, it works, and yes - we teach this in our Data Vault Modeling and Certification course.

I'd love to hear your thoughts and comments.

Daniel Linstedt

Posted August 26, 2008 5:32 AM
Permalink | 1 Comment |

1 Comment

Dan’s latest “secret” is a great example of a real-world tactic that leads to practical results, even if it may not jive with theoretical project implementation best practices. The principle of showing the business exactly how bad the raw data is, as a means of uncovering the real issues and workarounds, is very powerful.

Another means of accomplishing this is through data profiling, where you are not yet cleansing data—you are merely analyzing the data as-is to begin uncovering all the data quality issues, both known and hidden. Over and over again, I have seen organizations which have been lackadaisical about data quality, struggling to garner data warehousing and management resources and business involvement, suddenly “wake up” when a business executive is shown even a glimpse of the underlying issues in their data structures and data quality. All of a sudden, resources are committed, projects are kicked off, and the business is engaged.

On the other hand, I believe there is another way to increase business involvement in building and enhancing the EDW. Instead of merely sitting in endless meetings with the business, “documenting” their requirements in word docs and spreadsheets, we need to empower the business, particularly the business analysts, with tools where they themselves can analyze the data and specify business rules themselves for analytics and decision support. For more on this topic, see my posting on IT-Business collaboration at

Leave a comment

Search this blog
Categories ›
Archives ›
Recent Entries ›