This blog has been inspired by Martijn Evers blogpost, by Barry Devlin's work on Business Integrated Insight, by some very forward thinking customers and is bugging me for some time now.

Back in 2005 I gave masterclasses 'Data Warehousing in Depth' at the knowledge institute CIBIT in the Netherlands. One part of the class was pondering about the future of data warehousing. It was my favorite part and I remember that I always reminded the class as to the root-reason for a data warehouse; a physical construct to overcome deficiencies like performance, history, auditability, data quality, usability, ...

And I remember drawing two circles; one for data in the operational environment and one for data in the informational environment. Both represented physical environments, in other words; the data needed to physically move from the operational to the informational environment.

What would be necessary to blend the two environments, I then asked;

•  "Sources need to maintain history and adhere to strict auditability rules"
• "More and faster hardware and parallelization options in both hard- and software"
• "A common vocabulary of data, reflected in the operational systems"
• "Enterprise-wide data like master- and reference data needs to be maintained pro-active (upstream) and centrally"
•  ..

These are all still very valid points, but I would add a more profound one:

Data Virtualization guru's preach to us the Information Hiding¹ pattern. A pattern perfectly suited for the decoupling information systems and their data. These Data Virtualization guys and gals say that the software that supports this data virtualization is the new pinnacle for decoupling operational and informational environments and make the Data Warehouse² (eventually) obsolete. 

My opinion; 'they' are right and they are wrong. Yes, Information hiding is a pattern (there are more) that enables decoupling of the information environment and the operational environment. But this distinction is somewhat flawed - it is a distinction that originated from the early 90's that reflected deficiencies of using registered data for decision support activities.

We might wanna make an effort in trying to get rid of these deficiencies....

In the last 20-30 years, the data-model and its instantiations (the data) where directly based on the Information systems. The data-model and the data were by-products. The data-model fitted the information system. The Informational environment was born to overcome the deficiencies that came with this approach.

I want to make a plea for shifting this process-oriented thinking and designing to data-oriented thinking and designing. Make the data smarter instead of making countless little/big information systems with their own 'data-store'. It is not that odd; information systems and the business processes they support are so much more susceptible to change than the data is. Data is - in its very nature - extremely stable over time.

This plea is not new; start with an Information Model of your business, construct a conceptual model and slowly design your way down to the logical, physical model and/or canonical model³. Information systems are now made to fit the data-architecture and not vice versa. These information systems are somewhat decoupled from the data architecture. These information systems need to use the Information Hiding pattern.

Now, the principle of Information Hiding and the technology that can handle this pattern can flourish. Data Virtualization technology can be used to its full potential, but the same applies to BPM technology or technology which is based on service oriented architectures.

So, now we have the following list of requirements that are necessary to blend operational- and information environments:

1.     A data-oriented design and architecture of information system as opposed to a process oriented design and architecture

2.     "Sources need to maintain history and adhere to strict auditability rules"

3.     "More and faster hardware and parallelization options in both hard- and software"

4.     "A common vocabulary of data, reflected in the operational systems"

5.     "Enterprise-wide data like master- and reference data needs to be maintained pro-active (upstream) and centrally" 

And I will add another three:

6.     Centralized design and maintenance of business rules

7.     Data security and privacy law and regulations are enforced on the data-level.

8.     An organizing framework for establishing strategy, objectives, and policies for (corporate) data⁴

It is impossible to be completely thorough in this list, there are indeed many more, but this is a blog.....not a book ;-)

The mentioned criteria can be mapped to a discipline that is about to reach a critical mass in terms of body of knowledge, technology support, rigor in science and relevance in practice; Data Management and Data Governance. 

Back in 2005, in my masterclasses, I mused over the future of data warehousing. The future is now, the journey will not be easy, but the rewards are substantial. It truly is the future of the Sense and Respond⁵ organization.

Footnotes:

¹ David L. Parnas, 1972, On the criteria to be used in decomposing Systems into modules

² Data Virtualization vendors often claim to make the Enterprise Data Warehouse obsolete (referencing to the Boulder BI Brain Trust meeting beginning of this year where a vendor made this claim). They confuse a technology (data virtualization software) with an architectural construct.

³ A Canonical model is a design pattern used to communicate between different data format. It needs to be based on the logical model of the organization.

⁴ Jill Dyche and Evan Levy, note from the blog-post-author; I adapted the quote by adding brackets between 'corporate'

⁵  Stephen H. Haeckel, 1999, Adaptive Enterprise: Creating and Leading 

There is something going on for some time now, decades even. It all started with the arrival of the Internet where people voluntarily contributed data to, well, everyone who was interested. Data about themselves, their relationships, their adventures, their careers. This data was shared with consent of the owner of the data - although not everyone knew what the data was used for. So one might say that there was consent, but not informed consent.

Lets take it a step further and imagine....

What if our data  - generated by others - was given back to us and we could consent in an informed manner, that we wish to share this data for the greater good? For example; my tax information is my data, it is about me and I want to decide whether or not others can use this data. Or, suppose I can get a hold ofmy location/GPS data, showing all my movements. Or my point of sale data from the grocery store, showing my eating patterns. Suppose I can even get a hold of the data of the last MRI I took, my genome data or the data of my last blood test or even the data of a clinical test I was in?

Imagine....

What if I could decide to contribute this data (consensually) for the public good, where my privacy was still being honoured? What if dozens of people would decide that? What if millions of people would decide that? Clinical research would never be the same again. We would be able to scan for patterns in seas of data consisting of environmental data and healthcare data. No more clinical trials with just 2000 people and ever-increasing smarter statistics. In this setting the healthcare specialists, the quants, the sociologists and the behavioural scientists would have an unprecedented test bed of data. Is there a correlation (or even causality) between aspects of travelling, career, eating patterns, social status and cancer? Suppose even several generations would contribute their data; what would that mean for clinical research? Mindblowing.... 

In the above I discussed data that was about myself and so I should be the one who should decide whether or not to share. But what about data that is ours? The government heavily sponsors research in many countries. Research on biology, behavioural science, economic science, climate science etc.. Shouldn't the data generated by this research be public domain? I think it should...

What about data created by government - which is us. Data about im- and export movements, data regarding employment, schooling, law enforcement, crime, etc.. 

Imagine....

What would this democratization of data mean with regard to innovation? I think it would truly ignite a burst of possibilities and a huge potential for our general wellbeing. And no - I am not referring to the challenge of marketing handbags to middle age ladies (quote somewhat paraphrased from Neil Raden).

No, set the data free to go for the real challenges we face; decreasing poverty, climate control, improving healthcare, scarcity of resources, economic stability and decreasing crime.

This blogpost is hugely inspired by John Wilbanks - google the guy (!) -, all the Open Data initiatives of the world where governmental agencies free up their data, the technological possibilities of data storage, data deployment, data enrichment, data visualization and advanced analytics and finally...this blog is inspired by a deeply felt wish and conviction that our field of knowledge (data management and data utilisation) can make a contribution to a better place for us to live in.

I have always been fascinated by the true origins of modern-day phrases or trends in my domain - Information Management, data management in particular. It is like a challenge I give to myself, a puzzle waiting to be solved. Why you say? Well, Aristotle said it already:

'If you would understand anything, observe its beginning and its development'.

I tend to collect first the modern-day writings about it, mostly by practioners. Then I go to the on-line science libraries and browse through ACM journals, MIS Quarterly journals, European Journal of Information Systems, IBM Systems Journal, Decision Support Systems, Journal of Management Information Systems and lately the journal of Data and Information Quality Research. And I am forgetting a whole lot. But, since the field of information management is a relatively young science, I tend to eventually end up in the more or less classic science domains; psychology, mathematics, engineering, etc.. If I took it any further I would probably end up with philosophy and Theology ;-) and discover the meaning of life.... 

Being on such a quest is like opening up an unprecedented series of presents given to me by brilliant men and women. There is so much out there that can easily be applied to other domains, for example, the information management domain.

With 'Data Quality' the same applied. I started with books of Thomas Redman, aka the data doc, of course Larry English, Danette McGilvray, David Loshin, Jack Olson and also Arkady Maydanchik can not be missed. And one cannot overlook the books written by Yang Lee, Richard Wang and Leo Pipino. The majority of these books however (with the exception of Lee, Wang and Pipino), lack the scientific rigor, the kind of Design Research approach as introduced by Alan Hevner in 2004 (published in MIS Quarterly). And although this type of research is relatively young, there are many scientific based papers out there that more or less adhere to several of the Design Research pre-requisites that aim to have scientific rigor and relevance in practice.

Since 2004 many papers on data quality have been published that are really precious to me, but it just was not good enough for me. I had not reached the true origins yet, so I felt. So I broadened the scope to 'Quality' in general. Quality in a manufacturing/engineering/services context pointed me in the direction of Shewhart, Demming, Juran, Crosby, Feigenbaum, Ishikawa, and also Peter Drucker. Boy - did I enjoy the writing of these guys (sorry, they were all men).

However, I slowly digressed into various domains that opened up Pandora's box; the domain of coping with change, management theory, decision theory, group processes, system theory, system dynamics and much more. And although I studied on a university, economics, this was all new to me. 

Still not sure whether I have not being paying attention back in college or my university just sucked.

In between I entered into the field of Quality Software Management, not that odd I would say; on an abstract level one might argue that it is the sum of the above combined with software engineering and my own professional domain and the projects I undertook. Back then (and I still do) I felt that Gerald (Jerry) Weinberg seemed to have captured the soul of all these quality people combined with system theory, system dynamics, software engineering, a profound human perspective and a keen view on leadership and management (and why many current management models simply disfunction).

If anyone want to really go on a quest regarding 'agile software development'; do not bother, start by reading the books of Jerry Weinberg. You will not find the word 'agile', but you will recognize it.

These books (and he wrote a whole lot) put me on a roller-coaster (which I am still on) that included exploratory testing, self-organizing teams, leadership, Kanban/Scrum/XP, CMMI, Six Sigma, etc...

I have so many books now, so many papers, so many subjects, so many loose ends.....it is ridiculous.

And it all started with 'data quality'....

Am I done yet?

Hell no

Will I ever be done?

Hell no

Is it fun?

Hell yes

I need a second life, and a third...