Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

February 2006 Archives

My last post generated some great responses, ranging from Master Data Management as MDM philosophy to MDM Data Marts as "better than gospel" or better than system of record on the source system. In this entry I will take a look at MDM in a little more detail, and try to answer my view points on some of the issues raised in the responses. First, thank-you to all those who are responding, I enjoy reading your thoughts.

There is an interesting twist to compliance in one of the responses I received, the fact that a Master Data Mart (as I would call it) is better than the System of Record data, and should be taken (and often is) as Gospel to the business. Let's delve into the pros and cons of calling a Master Data Mart (altered, integrated, cleansed data) Gospel to the business. These by no means are complete lists, just a few thoughts running through my mind.

Pros:
1. Better information quality (in some cases) - but who's to say the Quality engine that is applying the rules, or the ETL/T rules are "better" than the data set that was actually entered? If a business changes data that I enter without checking with me first, I consider that an offensive move - why? Because the next time I contact the business they insist that their information is right, and I am wrong. Talk about loss of customer loyalty! This is just one of things we have to watch out for. Who's judging Better or Worse?
2. Single view of the enterprise data set - can't argue with that, there definitely is value to having single consolidated lists of data that all say the same thing... What we have to watch out for here is the security and governance of the data set - who's using it, when, where, and how are they applying it? Is it a good thing? From an IT perspective it's a great thing, when used properly in business it's also a great thing.
3. Lower cost to manage - with consolidation, comes a single store of information centered around business - so it is possible to have a lower overall cost of management of the systems, the data stores, and the routines that deliver the data.
4. Ease of implementation of EII, and SOA - with consolidation comes a necessity to implement metadata, and metadata management, and with that comes a level of understanding that assists in the build out of consistent EII and SOA systems.

Cons:
1. Too much of a "Gospel" thing - watch out! Altered data is NOT necessarily always good, I've got some examples of the insurance industry taking my information, interpreting it - they way they think it should be, then abusing it, in marketing, as well as claim cases. Besides that, someone who knows the consolidation routines can alter the data sets in a way like you've never seen. Most of us though are not bad apples by default, so this would never happen right?
2. Beware of the Business User shirking the system - if there is a way, there will be individuals who will find the loop-holes. With certain consolidations, loss of detail occurs - with a loss of detail and a merge, there are bound to be loop-holes in the system, ways in which certain individuals can learn the patterns, and overcome the "edit checks" or consolidation routines. But of course, we have error marts, and metrics to monitor these problems right?
3. Gospel doesn't mean "Audtable" - just because the business thinks the data is better, doesn't necessarily mean it's auditable or can be called a warehouse. The consolidated data often looses detail information, and traceability. In doing so, the auditors can (and have) take the company to court. Remember that implementation experts in IT are the ones who built the routines, and the business is never wrong... I've seen it happen first hand.
4. it’s just as important to know what's BROKEN as it is to gain a consistent view of the business information. I don't object to Master Data Marts, in fact, I think they're a great thing. I object to the loss of detail - at a data warehousing level. I've saved a specific company over $45 million USD in 6 months through a stringent warehouse process, yes we produced MDM over 10 years ago, but it was for data marts. By the way, we also produced a Master Error List (error mart) in which the business (for the first time in 15 years) began FIXING the broken business processes in the source systems. Why? Because they finally began to have visibility into what it was costing them to have "broken" operational systems - all because we tracked errors on a detailed and auditable level.

I believe that MDM and Master Data Marts are a wonderful thing, they do have value in business - and I highly recommend undertaking these efforts. However I still do NOT believe they should be a part of an Auditable Data Warehouse where detailed level data should exist and remain traceable. Also let's take a look at Master Data Management, and what it means....

MDM has the phrase "Data Management" embedded within. IBM just announced $41 Billion investment to be in the Data Management Space (or some huge $$ figure like that). Data Management is absolutely grand, wide vision - ultra-wide and multi-spectrum. Before you can "master" data management, you must master the data management side of the house. Maybe the acronym isn't so good after-all? In another post we'll explore just what Data Management means (at least from the Web's point of view). I'll also come back to physically federated data marts (MDM style), versus consolidated Master Data Marts (in a single physical location), and discuss the pros and cons of these aspects.

Feel free to weigh in with your pros' and con's - if we can build a substantial list of pro's and con's for MDM as Gospel, I'll write an article dedicated to it, and list all the sources. I think it would be good for everyone to have a say in this.

Thanks, see you next time,
Dan L


Posted February 28, 2006 5:35 AM
Permalink | 2 Comments |

Ok, we've all heard the term: MDM or master Data Management, but what the heck does it mean? My opinion is different than most, and in my search for the ultimate compliant warehouse I constantly battle with new acronyms... What EXACTLY is MDM? What about CDI (customer data integration)? Ok, how about these: PDM, BAM, BPR, LI, CRM, ERP, SEI, CMM, PMP, HIPPA, SARBOX, and so on? Well, I went overboard again (oh happy days...) Acronym soup is nothing new to me, I've been in the industry for over 15 years, been through the 80's and the Business Process Reengineering (BPR), also known as Lean Initiatives (LI), Six Sigma quality improvements, and ISO (international Standards Organization) setup.

Now comes BAM (business Activity management), MDM (master Data Management), CDI (customer Data Integration), PDM (parts data management), SCM (supply chain management), CML (customer Master Lists), and so on... Maybe one day we will realize that some of these acronyms are just new names for OLD (but valid) business goals. In this case, it's all about the business management, better quality, and shorter lead times, reduced overhead and increased revenue - after all, if I'm not making money, then why am I in business? In this entry I will focus on MDM and explore just what it is.

MDM aka: Master Data Management is likened to the production of "master lists" of centralized data sets. The notion that these data sets are "gospel" to the enterprise, and are agreed upon all the way to the top (board of directors), that these are single consolidated lists of data based on business keys, and descriptions only furthers the effort to use them as THE MASTER, the GOLDEN KEY, the LIST of: parts, services, products, deliverables, people, customers... you name it, there can be a master list for it....

BUT MDM takes it seriously as GOSPEL. In other words, many organizations and executive decision makers take an MDM to mean: it's the only SOURCE of all the integrated data of that type (CDI (customer for example)) that exists within the organization. The executives have just turned an MDM into a SYSTEM OF RECORD, like it or not, it's now got to be compliant with the WAY the business does business. It's a centralized system of data that has been cleansed, integrated, and archived. By the way, when completed, it also feeds 90% of the source systems from it's Master List.

BUT WAIT!!!!! What is an MDM really? And if I change the data on the way in, is it really compliant?
Here is where the discrepancies pop up. If the corporation were ever to be audited on the MDM they've built, they might be surprised to find out that keeping it as an audit trail much less a compliant audit trail will become extremely difficult, and in fact - won't stand up in court. Why? because the data itself within the MDM is essentially FLAWED, it has been altered to "meet today's expectations of what the TRUTH is." (and you know my feelings about the TRUTH, truth is all subjective no matter how you look at it.)

Now hold on, I'm not saying there's _no_ intrinsic value to the business to strive towards an MDM... I'm actually questioning if it really deserves the term MASTER DATA MANAGEMENT, I prefer to think of it as MASTER DATA MART - why? because my definition of data warehouse has changed. My definition of Data Warehouse (which is truly MASTER DATA) is essentially integrated, consistent, and EXACT copies of data as it stood on the source systems. (I define this much better in the Data Vault data model architecture at: http://www.DanLinstedt.com)

In other words, there are such things as Master Key Lists (MKL), and a horizontally integrated (without changed) data sets across the enterprise, but in my mind, the minute we put together CHANGES (quality, cleansing, profiling, aggregation, merging, purging, etc) in the data set, we've effectively produced a Data Mart. CHANGES TO THE DATA MADE BY ANY PROGRAMMATIC SYSTEM WITHIN THE WAREHOUSE CONSTITUTE THE CREATION OF A DATA MART, in which data is seen by, and created for specific user purposes.

Hence, I would like to say that MDM is really: master Data Mart, and enterprise accepted view of the data, that is not necessarily auditable nor traceable without the "warehouse" to back it up, in other words, the warehouse data to describe HOW we got to this level of integration.

Hold on, does MDM have value to the business?
ABSOLUTELY. There's no question that an MDM (master data mart) has value to the business, or we wouldn't see things like master parts lists, master customer lists, master services, master employees, master suppliers, etc... There is definitely value in consolidation and cleansing of data FROM the warehouse TO the data marts - there is definately value in producing data marts to meet the enterprise needs, answer questions, and provide a basis of historical data for pattern searching. But please don't confuse MDM with Data Warehouse, MDM is a data mart even if it's called Master Data Management, the end result of building one of these should be seen as, viewed as, a data mart and nothing more.

Now, can data marts feed operational systems?
Yes, as long as the physical source data from whence it was computed is actually available, auditable, and complaint. I can't stress enough how Data Warehouses must be compliant. The only time a Data Warehouse must not be compliant or probably does not need this level of source detail is when the warehouse is used for Strategic purposes, is ONE WAY IN (data comes in, is cleansed, and integrated, altered) and NEVER feeds source systems, and NEVER feeds operational activities and NEVER is involved in operational decisions.

CDI (customer data integration) is a form of MDM, and is just another data mart, and if you treat it this way - and keep your warehouse compliant and auditable, you are bound to have huge success moving forward. Contact me for success stories I've assisted with in the industry, I'd be happy to talk to you about this whole notion.

I'd also love to hear from you, even if you disagree, or have other opinions on what MDM means to you. Please elaborate below, all comments are welcome.

Thank-you very kindly,
Dan L


Posted February 7, 2006 10:03 AM
Permalink | 8 Comments |