Wednesday November 16^th 2011 Ralph Hughes from Ceregenics was in the Netherlands. Ralph is author of the book 'Agile Data Warehousing: Delivering World-Class Business Intelligence Systems Using Scrum and XP'. Ralph is currently under contract to write more books on the topic of agility in data warehouse development.

I had been in contact with Ralph for some time; he wanted to know more about data vault, getting the facts, how it is actually used, what customers use it, how they develop and deploy, how it contributes to agility and how it impacted the business.

Of course, anything can be explained in writing or conceptually, but the 'real proof of the pudding, is in the eating'. Opportunity knocked when Ralph was in the Netherlands for his TDWI course on Agile data warehousing. He asked me whether or not I could arrange some customer visits in Amsterdam. Customers that use and deploy Data Vault and have attained a high agree of agility.

Tom Breur and me were hosts for Ralph and we visited the Free University (client of mine) and BinckBank (client of Tom), both in Amsterdam. Hans Hultgren (Genesee Academy) happened to be in the Netherlands that week and joined us as well. We met with both management and technical team members of the university and BinckBank.

Both clients were particularly interesting because their data warehouses are in production and in a mode of constant change. Both clients showed a remarkable predictability and reliability in coping with these changes. Change equated to 'business as usual'. I remember Ralph asking an engineer 'how long does it take to deploy a new data element to the warehouse?' The engineer replied: 'do you want to know the lead-time including my coffee break?'.

Ralph, Tom, me and Hans were impressed with the accomplishments of these clients in getting their data warehouse deployment in control while constantly adding value/changes to the business in a predictable fashion. 

I will not transcribe the whole interview in this blog - that is simply too much - send me a note if you want to know more. Interesting differences between Free University and BinckBank were the fact that they used different automating techniques and also the level of business key integration differed slightly. Free University used templating (generating XML and import in Business Objects Data Services) for data warehouse automation and the data warehouse was driven by business keys. BinckBank used Quipu for data warehouse automation and the data warehouse was partly driven by business key, and some by surrogate key (see also my presentation on the Data Vault advanced seminar about different Data Vault species). In terms of software development methods, BinckBank used the Scrum method and Free University was based on waterfall/iterative with lots of lean practises being used.

I will try to summarize both visits from the perspective of me and Tom, particularly slanted towards Agile software development, by asking my blog readers, three questions: 

1. Why is it that you can build and deploy extremely small particles in Data Vault and not in other approaches, without having an increase in the overhead and coordination of these particles? In other words; 'Divide and Conquer to beat the Size / Complexity Dynamic'¹
2. Why is it that you can re-engineer your existing model and guarantee that the changes remain local? Something that is hugely beneficial in data warehouses that - by definition - grow over time.
3. Why is it that - as your (Data Vault based) data warehouse grows - your costs grow 'merely' in linear fashion initially, and as you approach the end state marginal growth in cost decreases exponentially (as opposed to exponential cost increase for Kimball warehouses)?

I want to thank Free University as well as BinckBank for offering their time, their energy and enthusiasm to the general cause of knowledge sharing. Of course I want to thank Tom Breur and Hans Hultgren for putting in their time as well. 

My special thanks of course to Ralph Hughes as being an open minded, inquisitive and knowledgeable peer. It was great being your host in the Netherlands. 

^{1 - Gerald M. Weinberg - Quality Software Management - 1992}

Photo #1: Left in the corner sits Ralph Hughes, next to him Tom Breur. On the other side the Free University; Jaap Roos (project manager), Dorien Heijting (Data Warehouse Engineer), Erwin Vreeman (Project Lead).

Photo #2: Sitting with the american flag - Ralph Hughes and Hans Hultgren. At the top of the table - BinkBank: Michel Uittenbogaard (Data Warehouse Engineer) and on the right Paul Delgman (BI manager). 

Photo #3: Sitting near the window looking down: me, myself and I