A plea for a more data-oriented approach towards design & architecture - Blog: Ronald Damhof

IQCP, a means to an end - it is all about quality

May 2011 I was given an opportunity to be part of the first group of pioneers that field-tested the IQCP program. IQCP stands for Information Quality Certified Professional and is organized by IAIDQ - the International association for Information Quality and Data Quality. To be fair; I really do not like certifications. It is said that with certification, clients can have some kind of objective measurement regarding knowledge and skill. Well, for the majority of certifications this is utterly deceiving.

That being said, why did I go for IQCP?

For one thing; IAIDQ is a not-for-profit organisation and vendor-neutral. People involved in IAIDQ are people with a passion to spread the word with regard to quality in general and Information Quality in particular.

Second; the certification process was extremely well prepared, grounded in known research and adheres to widely accepted standards and regulations such as the ISO/IEC 17024 and the Standards for the Accreditation of Certification Programs, published by the USA National Commission for Certifying Agencies (NCCA, 2002).

Third; I was not mainly interested in the certification as such....To put in poetic; I was interested in the journey, not the destination. The IQCP certification is based on a reference list of books and articles,carefully selected by people who know their stuff. The certification was a 'carrot on a stick' to me, pressuring me to actually study and invest time. The deadline was end of July 2011 - I have never in my life read so many books, but I have never in my life been so authentically interested in a subject as well. I just kept reading, because there is so much clever stuff written out there.

Fourth and maybe the most important reason; in may 2011 I also happened to follow the legendary PSL (problem solving leadership) workshop given by Jerry Weinberg,Esther Derby and Johanna Rothman. What I have learned is immense and influences me daily. The root of this workshop is - in my opinion - founded in quality principles; learning, continuous improvement, root-causes, self organizing team, people, communication, understanding the problem, understanding context and so much more. The people I met in this workshop were incredible and a huge inspiration. A lot of the attendees came from the world of context-driven testing and agile software development. Both - in my opinion - strongly rooted in quality principles.

It somehow all got together when I was asked to participate in the first batch of people to go for IQCP.

For me; IQCP was the means to an end - I have studied amazing books (for a reference list download this pdf), written by people like Edwards Deming, Joseph Duran, Jerry Weinberg, Kaoru Ishikawa, Masaaki Imai, Richard Wang and so many more. My library has grown very rapidly lately and I have still so much more to read (learn). Somehow, all I have read, learned and still learning are pieces of a puzzle that fit seamlessly in my field of expertise #datamanagement #decisionsupport #datagovernance #dataquality #datavault #architecture #softwaredevelopment.

The journey never ends....

My gratitude goes to Tom Breur, IAIDQ, Jerry Weinberg, Esther Derby, Johanna Rothman, those crazy canadians ('it is what it is'), Testsidestory, Olav, Markus, Griffin and so many more.

ps. I passed the exam.....I am now an Information Quality Certified professional

Posted September 21, 2011 3:33 AM
Permalink | 1 Comment |

Data Vault Introductions - Download

I get a lot of questions of people (especially in the United States) about Data Vault. For those who want to familiarize themselves with Data Vault, I hereby offer the links to some papers I wrote (sometimes with others) and a presentation I gave recently (see other post for details):

Published originally in Dutch for Database Magazine, an article that is the first part of a triptych regarding the 'Next generation Enterprise Data Warehousing', this part was an introduction into Data Vault named 'Letting go of the idea of a single version of the truth' - August 2008 - Download

Published originally in Dutch for Database Magazine, an article that is the second part of a triptych regarding the 'Next generation Enterprise Data Warehousing', this part deals with the post-Data Vault processing, the business rules in particular - November 2008 - Download

Published originally in Dutch for Database Magazine, an article that is the final part of a triptych regarding the 'Next generation Enterprise Data Warehousing', this part deals with 'Development processes in Data Warehouse environments' - June 2009 - link

Published in Belgium for BI-community.org, a keynote article titled: Data Vault, Business Objectives for next generation data warehousing - January 2011 - Download

And finally a link to the presentation I held on the Advanced Data Vault seminar, May 5 & 6 in Baarn, the Netherlands: link

**July 2nd,2011 - Tom Breur wrote a good piece "Tom's ten data tips", very much Data Vault related.

Posted September 21, 2011 3:29 AM
Permalink | No Comments |

For the love of Data (Quality)

"Cleaning the lake or reducing the pollution from the factory" - is an analogy used by Thomas Redman. It perfectly paints a picture of data quality issues 'we' all face in our data management projects.

In projects we often have to struggle against forces that 'just wanna create the freekin report'. Whether the data is wrong is of no concern. In these instances the goal apparently is the information system ('our DWH is running', 'The report is build' or 'SAP is live') and not the data. Put in other words; data is often treated as a by- product and the information system is the main product.

Lets take a closer look - by using (among others) Richard Wang's analogy with a manufacturing process¹:

I have never seen any manager, CEO or foreman happy with a successful implemented assembly line but a lousy product. Have you? In software engineering I sometimes have the feeling we lost touch....

The Information System - be it a Data Warehouse, a report or an ERP - is not the purpose, it is a means to an end. And the end should at least be sufficient data quality (where data quality is defined in the perpective of the customer, fit for his/her task).

The cool thing about the (somewhat) oversimplified analogy of Richard Y Wang is its usefulness for another reason. It stresses the system perspective you gotta have on dealing with data quality issues. You cannot go about 'cleaning your lake while the factory is still polluting'. Producing quality Information Products is executed by means of a system. Now, do not translate this 'system' into 'information system'. This system consists of people, processes and technology. Dealing with data quality issues requires a system perspective to really add value in terms of better quality products and a 'greener' environment.

So - do not blame your ERP department for creating bad data

So - do not blame your report builder for creating useless reports

So - do not blame the person entering the data

Maybe, something to consider; who do you think is accountable for organizing the 'system'? Yes - management should embrace quality in its DNA...

I know I am corny - management should have read Deming, Juran, Crosby in their MBA's. Knowledge that is like half a century old.

^{1 - Richard Wang - A Product perspective on Total Data Quality Management - feb.1998, Communications of the ACM}

Posted September 21, 2011 3:26 AM
Permalink | 1 Comment |

We need to be flexible - Do we? Really?

In 1967 Thompson wrote about the administrative paradox; a dichotomy where continuity (stability) and flexibility are positioned at both ends of the spectrum. In other words; be flexible and at the same time try to progressively eliminate or absorb uncertainty. This paradox can also be discussed in terms of time; in the short run administration seeks to reduce uncertainty. In the long run, the administration strives for flexibility.

Nothing new I hope? Now, what about Information Systems...

In using Information Systems we also need to deal with this paradox. We tend to use Information systems to automate tasks, formalize sequences of events, kill flexibility (;-)). An Information System can be interpreted as a 'bureaucrat in an electronic version'(Checkland and Holwell, 1998).

So, what do we do? We tend to modularize information systems, integrate them via services that are of course strongly decoupled with each other. IT delivers and supports all kinds of business functions and with a brilliant Service Oriented Architecture we cross the bridge between function and business process. We can now change the business processes if demand requires it.

Yee - we=happy. we=flexible again. Easy huh?

NO, it is not easy. It can be an open check you write to your 'partners' - the System Integrators, it may takes years before you capitalize on the investment that has been made. And in the process you tend to demotivate your own personnel (or customers) big time

My point; the balance between stability and flexibility is sometimes totally lost in organizations. Some architects and many vendors/solution providers are pushing the flexibility agenda big time nowadays, but the 'why' of flexibility has never been fundamentally discussed with(in) top management. The 'why' should be related to the industry your in and the strategy you wish to approach the market with. For example; I believe firmly that many government agencies should focus on stability over flexibility. But unfortunately, they seem not to agree with me. And what also needs to be considered is that stability and flexibility are interconnected; more focus on flexibility will diminish your stability and vice versa. Accept collateral damage if you architecture is all centered around 'being flexible', if you want both, well, you can not and expect to pay a price ;-)

Even if the case for flexibility is made, the 'how' should be extremely careful considered. Is flexibility in business processes needed (hard question)? Or is flexibility in data sufficient (which is a huge difference in terms of attainability, costs and organizational impact), but the latter can overcome the Administrative Paradox at least partly....

Posted September 27, 2010 11:59 PM
Permalink | 1 Comment |

Change always comes bearing gifts

A story.....

Vendor X sells its ERP to a company in Healthcare;
Client wishes to setup its informational environment (data sharing, BI, CPM etc..) right from the start;
Vendor X pushes the 'standard' solution' they sell;
Client decides to decouple their informational environment from its source(s) for several reasons (heterogeneous sources, sustainability, compliance, adaptability etc..);
Vendor X deploys their ERP;
Client starts to design and build the informational environment;
Interfaces between ERP of vendor X and the informational environment are developed;
The ERP of vendor X off does not offer functional interfaces ('X keeps pushing their standard product'), so client needs to connect on the physical level;
Going-live is near; of both the ERP and the new informational environment

And then change management of vendor X regarding the ERP kicks in.

Client: 'What's your release schedule for patches'?

X: 'Every 2 weeks'

Client: 'Huh'?

Client thinks: 'Damn, how can I keep up with this change schedule?'

Client: 'Well, can you tell me anything regarding the average impact of these patches?'

X: 'Well, they can be very small and very big'

Client thinks: 'Ok, what are you NOT telling me'

Client:'Ok, but this ERP is like 15 years old, so give me an overview of the average impact'

X: 'Basically anything can happen'

Client thinks: 'o, o'

Client: 'Ok, but the majority of these changes are of course situated in the application layer, not the data layer?'

X: 'Well..anything can happen.'

Client thinks: 'Is it warm in here?'

Client: 'Anything? Also in the data layer? Table changes, integrity changes, domain type changes, value changes?'

X: 'Aye'

Client thinks: 'Ok - I'm dead'

Client: '...at least tell me that existing structures always remain intact and the data remains to be auditable - extent instead of replace for example'

X: 'Huh'?

Client thinks: 'Well, at least I am healthy...'

Client: 'hmm...just a side note, we use Change Data Capture, I assume that these changes are fully logged?'

X: 'Nah - log is turned off, otherwise we can't deploy the changes'

Client thinks: '..hmm....is my resume up to date?'

My point; do not assume your vendor (of any system) to engage in professional application development and a change management policy that takes into account the simple fact that data of these information systems need to be shared with other information systems in your company.

Change management and professional application development needs to be important criteria regarding the selection of information systems.

Posted June 8, 2010 2:29 PM
Permalink | No Comments |

PREV 1 2 NEXT

Blog: Ronald Damhof

IQCP, a means to an end - it is all about quality

Data Vault Introductions - Download

For the love of Data (Quality)

We need to be flexible - Do we? Really?

Change always comes bearing gifts

Search this blog

Categories ›

Archives ›

Recent Entries ›