I find it interesting how companies tend to focus on the hot issue of the day—and then fire up ad hoc teams of available resources to attack the immediate problem. For those of us who have had a long-term career focus on enterprise information management, this behavior has been a recurring source of frustration. This is because many of the problems being addressed in this manner are the result of not having good data management practices in place. Not only that, but each one of these ad hoc projects typically compounds the data management problems in a never-ending cycle. Let me explain.

The ”best” example I have seen of this was working for a company that had embraced agile development. Every time there was a request from the business for new systems capability, the agile team would take great pride in building something quickly in response to the business demand. The business was initially happy that IT was responsive to their needs, and IT was proud of the statistics that showed they had delivered 240 new applications in the prior year. This was a very large company, so the number of systems is not an exaggeration.

The problem was that to sustain this pace of delivery, there was no attempt to engage enterprise architecture—let alone analyze information requirements against existing data sources. The result was that in the year mentioned, nearly 240 new SQL Server databases got built to support the 240 new application systems. After some time, the business realized there was some redundancy in these applications when they began to use them to report operational statistics and corporate performance indicators. They began to realize there were irreconcilable differences between these databases due to semantic inconsistencies. The application owners began to bicker over the differences and ended up divorcing themselves of each other. This resulted in fire drills every time a VP demanded that someone ”just give me the numbers.” This wasn’t easy since they weren’t talking to each other anymore. Ad hoc queries, Access databases and spreadsheets began to spread like wildfire throughout the organization.

Over time, nobody knew what was where, as the ad hoc teams that had been put together for each of these fire drills had moved onto other things once the immediate problem was resolved.

Businesses are no different than individuals in their behavior.   If someone tells us we should plan for medical emergencies or, heaven forbid, our own demise by saving or acquiring adequate insurance, we may put this off because we have more immediate pressures from our family to purchase a new car or buy a bigger home. These are things we can realize immediate gratification from—not something that may happen in the foggy distant future. The problem becomes exacerbated when we not only put off investments to provide for our financial security, but we also mortgage our future by acquiring debt to satisfy our immediate personal and family desires.

In the company I talked about above, that’s what was happening. They were not only not investing in data management practices that would have curtailed the proliferation of standalone application databases, but they were also incurring a growing debt in terms of the cost and effort that, at some point in the future, would have to be accounted for. The ad hoc reporting currently taking place was becoming increasingly unsustainable. This was recognized and an enterprise data warehouse was seen as a more strategic solution to consolidated reporting in the long-term. However, they kept digging a bigger hole in terms of the number of source systems and mapping complexity that would need to be dealt with to build an EDW. Also, with every ad hoc solution came more Access databases and spreadsheets that took on a life of their own and further compounded the data management issues.

There is one significant difference, however, between our personal lives and business. Most of us are taken care of to a certain extent by our employer’s benefits plans and government programs—such as social security that automates these savings and would otherwise require us to make a conscious effort to not participate in these programs. This is not the case in business. In business, there is stiff competition for every cent to be spent that mandates the development of a strong business case if you hope to acquire the funding you need for your purposes. So rather than the relatively passive approach we take in our personal lives to ensure our long-term financial security, there is a need to aggressively secure funding to reduce our businesses data management risks.

So if you’d like to improve data management at your company, but have been frustrated by the lack of authority, resources, or company politics, there are a couple of things you can do:

• Make a business case—so they’ll give you the money.  
• Institutionalize data management—so it becomes like a benefits program that just keeps getting contributed to.

The most success I have had in addressing these two recommendations centers on cost justifying an Enterprise Information Management Competency Center (EIMCC) and embedding its roles into the company’s management, systems development, maintenance and compliance processes.   The following is the approach I successfully used at one company to do this:

• First, I analyzed the costs of the data management work (data architecture, data analysis, data modeling, DB design & support, ETL, data integration, information delivery, data policies and access, etc.) that had been done for all significant projects over the last couple of years. This provided heuristics to use for projecting the costs of this work against all future projects.
• I then analyzed the redundant data infrastructure and databases that had been created and were continuing to grow.

Using this information, I was able to develop a business case to demonstrate that had the company implemented an EIMCC, the costs for these activities and infrastructure would have been approximately 50% less than had been incurred. The main reason for the reduction in costs would be that data infrastructure and data content would have been leveraged across multiple projects instead of duplicated.   By using EIMCC resources on each of these projects, they would have also executed more efficiently because the specialized resources would already have the core data knowledge, designs and metadata available to them to support the new initiatives. The business case also addressed compliance by identifying data governance and compliance accountability and embedding the processes for these into the SDLC.

After implementing the EIMCC and institutionalizing the appropriate management disciplines and development processes, the cost of these activities after two years had dropped to 30% of what they were prior to having the EIMCC in place.   Some new resources had to be hired with the appropriate skills; however, these were more than offset by reductions of staff, primarily in the application development and maintenance groups.

Hopefully, you can use what’s been said here to help you come up with a successful strategy for your business and get beyond your data management frustration.   I’d love to hear from you about what approaches you have tried and what has and hasn’t worked.

photo by Daniel Voyager via Flickr (Creative Commons License)