Originally published November 7, 2007
Second chances are rather rare and should be used wisely. The data management profession now seems to be in receipt of one, and it is incumbent on us all to make a success of it. To do so requires an honest appraisal of the past and development of a vision to meet the challenges that lie before us.
The first step is to dispose of the term “data management,” which has been used here only by way of introduction since there is a broad understanding of what it means. The term “data administration” is an even bigger deadweight on the profession and has to go too. Their replacement, and hopefully it will be more than just a change in terminology, seems to be “enterprise information management” (EIM). Before we discuss what that entails, we need to learn some lessons from history.
“Data management” and “data administration” were synonyms for a range of activities that were common in the 1980s and 1990s. Let us use “data administration” since it better captures the spirit of what these activities were about.
Data administration rode the back of tools and methodologies that emerged from the relational database revolution that began in the 1970s. Relational database technology was incredibly successful in providing value to enterprises that adopted it. Tools and methodologies supporting the relational revolution appeared, and legions of consultants were spawned. Soon, an even greater promise was made. Not only could individual databases be designed, but the entire set of information assets of an enterprise could be described in a single data model. This corporate data model (CDM) would provide a single road map for all database designs, would prevent needless redundancy of databases, would of itself achieve data integration, would represent all data as the enterprise truly saw it, and a whole lot of other good things too.
Up to this point, there had only been successes, so data administration units were set up to build out these corporate data models. Even where CDM’s were not the goal, there were similar objectives. One was to standardize and reuse the components of data models intended for individual databases. Another was to coordinate such data models to facilitate data exchange among applications. The requirement for knowledge management was also recognized, and the goal of providing understanding of the data an enterprise possessed was handed to data administration. It was also during this period that data began to be recognized as more than a by-product of applications, and something of real value in itself. Data warehousing began to take form and begin its steady ascent.
Data administration enjoyed quite a long time in the sun. However, just after the new century was ushered in, it ran into problems. Several major events occurred, but perhaps the most significant was the bursting of the Internet bubble, which happened in March 2000.
When the bubble burst, there was a general feeling that a good deal too much faith had been placed in IT generally, and that some IT people had been less than honest in describing what they could actually deliver. And, of course, a lot of money had been lost in the process. The “brick and mortar” business folks, who had been portrayed as Neanderthals facing extinction during these excesses, were in no mood to invest a cent further in IT after the meltdown occurred. Soon, the sharp blades of cost-cutting exercises were felt in IT departments. The tragedy of 9/11 only provided further momentum to these efforts, and there was a sense of urgency as a real economic downturn was feared and businesses wanted to be prepared to weather it. It was during this period of cost-cutting that the lure of offshoring proved irresistible, and huge amounts of IT work were transferred to India and other places.
The “dark age” lasted from 2000 until roughly 2005. It was during this time that many organizations drastically reduced, or eliminated, their data administration units. There were warnings of doom from the affected data administrators, and perhaps management did genuinely worry about what might happen. But nothing did.
The main reason that data administration units got shut down was that there was a perception that they did not deliver much of value.
A primary problem was that data modelers viewed themselves as creating logical data models. These were not database designs, although they could be the basis for them. This set a terrible precedent of dividing the world of data into logical and physical camps that interacted very rarely. Data modelers did data models, and the developers did their thing. It was rare that implemented databases looked anything like the logical data models they were supposed to be based on.
Meanwhile, work on the behemoth corporate data models continued. One problem with a data model is that it is a gigantic batch container. If you treat it as a single thing that has to be made ready, then you have to wait quite some time. Another problem with data models is that they are poor mechanisms for knowledge management. You have to be a data modeler to really get knowledge out of them. They are created in expensive tools that have restricted numbers of licenses in any enterprise, meaning the tools are accessible to only a few in IT and nobody in the business. All of this made data administrators look like priests and priestesses of the cult of Isis guarding sacred scrolls in some inaccessible temple.
A particularly sad consequence of corporate data models was the search for single definitions of entities and attributes across the enterprise. It was felt that there must be just one definition of, say, Customer, and if we knew what it was, many benefits would flow. Data administration embarked on this quest, and facilitated interdepartmental strife between areas like marketing, who insisted Customer had to include prospects, and accounts receivable, for whom a Customer is someone who needs to pay a bill. It was never clear just why data administration decided there had to be a single version of the truth for everything, other than it would be convenient for data administration.
None of the above may have been specifically articulated by senior management as they closed down data administration units in the “dark age” of 2000 – 2005, but it pretty much guaranteed the demise of data administration.
The strange thing is that demand never went away. If data was always confined to silos, nobody would bother about it very much. Yet we know that it has to be shared, exchanged, and integrated to
make modern enterprises work. When this happens, data is recognized as having value. It is needed to manage enterprises now that there are so few middle managers left. It is the key to
understanding how profitable, efficient, or risky business truly is. It is a competitive advantage. It is gold dust.
So here we are again. The emphasis is now on making data sharing and exchange work, and unlocking the value of data. If EIM is to be successful, we need to avoid making the mistakes of the past. The first thing that has to go is the divide between the logical and physical worlds. EIM must span both since it is an inconvenient fact that data is ultimately physical values. The next thing that must be gotten rid of is the notion that data modeling is the sum total of everything EIM does. It has an important place, but it is not everything. Enterprise information architecture, information knowledge management, data governance, master data management, and so on are not primarily data modeling tasks. Retreat into the comfort zone of data modeling concepts in the logical layer using tools nobody else can understand will be self-defeating.
To succeed, EIM will really have to align itself with the business, as well as IT, and deliver recognized incremental value at frequent intervals. It will have to be outward looking, oriented to the business, not inward looking and oriented to data theory. None of this will be easy, and it is quite likely to be messy. However, we should be grateful that enterprises have realized the need to understand, control, and derive value from their data assets. My Enterprise Information Expert Channel for the Business Intelligence Network and my articles are dedicated to understanding the new paradigm and finding practical ways to make EIM work. Viva enterprise information management!
Recent articles by Malcolm Chisholm