Business Intelligence Network - The Vision for BI and Beyond

IQ in the News
Most people have probably heard about the highly reputable Consumer Reports’ recall of its flawed testing of infant car seat safety. The report, issued January 5, 2007, found that many car seats failed the high-speed side impact test it conducted (the government requires passing of frontal crashes at 30 mph). Consumer Reports tested at 35 mph for frontal crashes and 38 mph (so they thought) in side impact crashes. The findings seemed to indicate a high degree of failure with nine seats failing some or all of the crash tests, and only two doing well in all tests.

However, the government found a problem with the way the testing was conducted. Instead of a 38 mph side crash, the test simulated a side-impact crash of more than 70 mph with results very inconsistent than what would have come from 38 mph tests. Consumer Reports recalled the entire report on January 18.

IQ Lessons Learned From the Consumer Reports Recall

• Negative impact on consumers and their confidence in the organization: The impacts of the faulty testing were dramatic and swift. The Executive Director of the Washington State Safety Restraint Coalition exclaimed that “Consumer Reports screwed up….They really upset people and created enormous confusion.”
• When designing tests, as you will with information quality (IQ) assessments, you must assure you design the tests properly. Validity and accuracy are two distinctly different measurements. You can test validity by defining the business rules, valid values or ranges the data must conform to, and conduct these tests electronically with IQ assessment software or your own validity routine tests. But to measure accuracy, you must confirm that the data values correctly correspond to the characteristic of the real world object or event the data represents. To perform this test, you must compare the data with the characteristic of the real world object itself. In the case of car seats, Consumer Reports believes, rightly as I believe, that crash tests should be conducted at high speeds, which are more representative of actual accident experience.
• When you make a mistake, own up to it and apologize for it. Then do everything you can to ameliorate the error and its impact. Consumer Reports retracted the report as soon as they determined the serious problem with the study. Jim Guest, President of Consumer Reports, wrote “A message to our readers” on the Consumer Reports home page, with important messages to his customers: “By now, you've probably heard the news about my decision to withdraw the infant car seat report featured on ConsumerReports.org and in the February issue of Consumer Reports magazine. I took this action when we discovered a mistake in our side-impact crash tests. We always strive to be accurate and fair, and I regret this error. Going forward, I want to make sure that our actions are as thorough and transparent as possible so that we preserve your trust as we continue to test, inform, and protect consumers.”
• When you have IQ problems, but must have accurate and complete data, you must pay the price of the process failure and the costs of “information scrap and rework.” Consumer reports is retesting all of the infant car seats to provide the comparable data.
• “Reputation” of an information provider is not a guarantee of the quality of information provided. Even the best make mistakes. One must error-proof processes based on root cause of failure. A better measure is the reliability of the processes to provide consistent, quality information based on the kinds of error-proofing provided and consistency of the process results.
• When you have a significant IQ problem, you must analyze the root cause(s) and improve the process to prevent the root cause(s) from causing failure again. Consumer Reports will be conducting extensive analysis as to what went wrong in these tests to assure they will not recur. This is the same approach we use when we find critical IQ problems. We must conduct root cause analysis, find the root causes and improve and verify the efficacy of the improvements to prevent defect recurrence.

What do you think? Let me hear at Larry.English@infoimpact.com.

If you found this article helpful and would like to receive the latest insights each quarter from Larry English and other experts featured on the Business Intelligence Network, please subscribe to the Information Quality Newsletter.