We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Podcasts

Some Perspectives on Quality


 

Originally published July 23, 2009


Overview

According to Bill Inmon, the notion of data quality takes some surprising twists when viewed in the context of data warehousing and decision support processing.

Bill Inmon
Bill Inmon

 
 

Comments

Want to post a comment? Login or become a member today!

Posted July 24, 2009 by Phil Bailey phil.bailey@btconnect.com

I agree with Karien, and also Bill, as one the first benefits to be achieved from a data mart or warehouse is VISIBILITY of the data from your business apps... and if IT think they can or should be accountable for 'fixing' it then I would argue that they should think again. IT are merely custodians of data, not the owners. Garbage in, garbage out - IT do not have a magic wand, and should not try to fix things they are not asked to fix by the business.

So George, surely your problem lies with your IT systems, although I appreciate that in complex, large scale environments, fixing small things like code value anomolies is difficult to get 'fixed' back in the source systems and you have to 'live with it' in the DW domain.

In my designs I try to provide a view of the RAW source data within a persistent version of the staging layer, (building up history that may no longer be stored in the sources over time due to archiving etc) and then providing the conformed, standardised and 'fixed' view within data models - but making sure that it is possible to reconcile back to the raw layer to allow users to trust that the data is correct.

If you were to 'fix' a null value in a particular column, how do you then reconcile this back to the source? Your reported figures are instantly different from the same count from the source system; creating one of the gravest problems in MI - inconsistnet figures - instantly your DW has lost it's credibility!

Is this comment inappropriate? Click here to flag this comment.

Posted July 24, 2009 by Karien Verhagen

A very important issue about data quality is the awareness of the business of the importance of the registration of correct data. This awareness is not served by hiding all kinds of correcting and cleaning algorithms in the ETL. Given the choice it is far better to report incorrect values and improbabilities to the source data owners and let them correct if appropriate. Source owners will develop awareness of their links with other lines of business and develop a corporate awareness that is very valuable and a lasting contribution to data quality in the corporate data warehouse.

Is this comment inappropriate? Click here to flag this comment.

Posted July 23, 2009 by george.allen@va.gov

I definitely fall into the Larry English camp.  Data quality problems speak to poor programming practices at the front end application, allowing for incorrect data to be entered beyond expected parameters or not allowing for verification at the point of entry.  Correcting these problems at the point of moving the data to the warehouse is a exacting process but also allows for more flexibility and accuracy of the data within.

Of course, if one is only expectign aggregation to come out of the warehouse, then small errors are of no consequence, but if the small error prevents a whole set of data from being included in the aggregrated result, say an improperly coded department number, then the errors become much more problematic.

I work in the healthcare arena, and have many data quality issues to deal with caused by too many data entry loopholes in the applications recording the information.  Though my warehouse is mainly used for aggregating care performance, we are now branching out to more detailed analysis of protocols and direct patient care costs, so we cannot allow for as many errors as we might with more aggregated reporting.

This is my personal opinion and not necessarily the opinion of the Department of Veterans Affairs.

Is this comment inappropriate? Click here to flag this comment.