Washington, DC on May 15, 2008
Chicago, IL on May 20, 2008
New York, NY on May 21, 2008

We will be focusing on how changing business requirements are creating new data quality challenges, and how we can adapt data quality solutions to help focus on providing the right data — in the right format, at the right time — to broad, global systems that drive strategic business goals.

Let me know if you are planning to attend and want to schedule some additional time to chat!

- MDM is more of a means than an end, and it is more likely to be justified in the context of other enterprise activities such as CRM or ERP.

- I have started to bristle at the phrase "golden copy." I now think that MDM is more about providing universal transparent access to a sngle representation of uniquely identifiable entity data, but that does not mean that entity data has to sit in its own silo.

- Comprehensive master metadata should include more than just data dictionary information

Stay tuned for more information on the book...

Two interesting aspects of the Spitzer situation. First, his tactics at using information to track down targets for prosection as NY State Attorney General are prime exmaples of exploiting business intelligence to identify patterns of misbehavior. Second, one would think that, knowing the tactics to be used to seek out suspicious activity, would have hesitated to expose himself to discovery via the same tactics.

I am finding, though, that I am building up an interesting set of books that provide value to the way I look at the use of information, so I thought I'd share a list of books that I have recently read, am currently reading, or plan to read some time in the near future. Each one deals with aspects of how we can learn from what we know, learn from what we don't know, then exploit what we can learn:

"The Wisdom of Crowds," by James Surowiecki
"Freakanomics, " by Steven Leavitt and Stephen Dubner
"The Tipping Point," by Malcolm Gladwell
"Blink," by Malcolm Gladwell
"The Black Swan," by Nassim Nicholas Taleb
"Fooled by Randomness," by Nassim Nicholas Taleb
"The Long Tail," by Chris Anderson
"Fortune's Formula," by William Poundstone
"Linked," by Albert-Laszlo Barabasi
"The World is Flat," by Thomas Friedman
"Collapse," by Jared Diamond

One of my roles at the event was to moderate a short workshop session to help attendees articulate what they believed were their most critical needs for master data management. One interesting common reaction was confusion about what composed an MDM solution, and what were the vendors actually selling. Another frequent reaction was expressing difficulty in lining up the requisite set of ducks within a reasonable amount of time to garner enough "horizontal support. Third, a general consensus was that instituting MDM was best done as an adjunct to existing application development (e.g. to support BI), focusing on small projects.

Actually, that last one confused me a bit, since if it only centering on a small application area (and not the whole enterprise), could it really be "master data" management?

Oh, one more thing - it may be worthwhile to consider the qualitative (and feasibility) differences between creating a "single golden source of truth" and an environment supporting the transparent access to a unified view of uniquely identifiable master objects (my current definition of what MDM is, by the way).

Orienting the data in a columnar manner is nicely suited to analytic applications, so the clear opportunities for these kinds of products are partnered solution providers for specific types of analytics, or with data aggregators and providers to allow for data linkage and then analysis.

Some of the vendors (or vendor reps) I bumped into over the past few days include ParAccel, Vertica, Sybase IQ, Infobright. Kognitio, alternatively, is not columnar but through data distribution across parallel systems can also talk the virtual appliance talk.

One conclusion that can be drawn is an emerging market for providing high performance analytics platforms with a low barrier to entry points towards cracking open that small.medium business market. One interesting thing to watch is the ways these guys will partner with other BI vendors (e.g., OLAP, visualization, end-user analytics) to see who can put together a robust end-to-end BI solution suitably priced for the $50-$100 million company.

Here is what I mean: using our business intelligence and data analysis and data mining and predictive analytics, we claim that we can increase response, reduce costs, extend customer lifetimes, improve lifetime values, etc. So as an experiment, I should be able to start a retail business and accumulate a bunch of customers who will always be satisfied, will never threaten to cancel their service, and will always be just about to buy the products I have already determined they need. They will each be at the center of a huge sphere of influence, and I will exploit the viral marketing opportunities by turning every satisfied customer into a walking advertisement for my products and services. I will have optimized my product and service offerings so that as one product becomes obsolete, the customer is dying to upgrade to the next level, and I will time their releases so that no follow-on product cannibalizes its predecessors' sales.

The idea intrigues me: pick a product or service to sell and then apply the performance improvement techniques driven by busiess intelligence. Some thoughts:

I would want to pick a business that is recession-proof (plumber? pest exterminator? funeral director?).

I would have to sell a product that needs updating or replacement within a relatively short cycle. Selling replacement windows is probably out. Selling office supplies is more like it.

You get the picture: a broad market where some knowledge of the customer community can drive repeatable sales, and where customer data is easy to get, maintain, enhance, analyze, and exploit.

There are a lot of success stories out there for applications of BI to business productivity improvement. Yet that is not true across the board, and that probably means that owning the software doesn't necessarily imply achieving the benefits without a little hard work. Ultimately, the successful organizations exploit BI by adapting their business processes to exploit the the knowledge discovered, and put practices in place to measure the value of each decision. Maybe that is what drives the belief in BI?

Yet the increase in adoption of tools doesn't necessarily mean that there is an increase in value; it would be interesting to see some report documenting how SMBs adapt the way they do business as a result of integrating BI into their operations.

However, the more I think about MDM and product data, the more intrigued I am. I have said this in a number of metings: product names are curious because they often describe what they are. For example, a PHILLIPS SCREWDRIVER 6-3/4" is a phillips screwdriver that is 6 and 3/4 inches long. What is more, product descriptions carry a lot of information that can be relatively easily parsed out using standard text analysis and text mining techniques. So I would very much be interested in hearing more about some product information MDM projects - email me or post your success stories!

"We recently sent you a letter in which we advertised a Verizon bundle package of Verizon FiOS Internet and Verizon FiOS TV service. This letter was mailed by mistake and the services described in the letter have never been offered by Verizon under those terms.

Verizon Consumer Marketing"

OK, seriously, I am finding it hard to get my head around this. The offer came in one of those pseudo-overnight envelopes that marketers often use to make their letter seem more credible - you know, cardboard weight with a zip-pull - not cheap. So this company:

- Drafts a marketing letter,
- Prints tens of thousands of copies,
- Custom prints tens of thousands of fancy cardboard envelopes,
- Puts them into fancy cardboard envelopes, and
- Mails them.

Actually, I am guessing about the number - it could be orders of magnitude greater, for all I know.

I find it hard to believe that the internal governance and control over marketing would not have stopped the process after the marketing letter had been drafted if it contained erroneous information, so I am curious as to what has really happened. I mean, in fact, having sent out the previous letter, the company actually did offer the services under those terms, but perhaps, due to some error, was not prepared to honor that offer.

In any event, my guess would be that there were some significant negative business impacts related to this bundle blunder - actual hard costs for materials and postage, as well as softer costs relating to organizational trust.

"Data doubles every year"
"The amount of stored data from corporations nearly doubles every year"
"...the amount of data stored by businesses doubles every year to 18 months."
"In his book “Simplicity,” business management expert and author Bill Jensen indicates that the most conservative estimates show business information doubling every three years, while some estimates say data doubles every year. "
"Unstructured data doubles every three months"

I am still following links from the first page of results, and we are doubling our data every 3 to 18 months.

"Reed's Law states that the volume of data doubles every 12 months. "

OK, so there is actually a law about it. Hold on a second, according to wikipedia this law is about the utility of (social) networks, so perhaps the law doesn't apply in all jurisdictions.

Anyway, these may all be references to a UC Berkeley study on the growth of data , which said that the amount of information stored on media such as hard disk drives doubled between 2000 and 2003.

So let's look at this a little more carefully - we have a scientific study that looks not at the creation of data, but rather the use of storage media to hold what is out there. And out there is a lot of stuff needing a lot of storage, like images, music, videos, etc. Things that have information yet from which are still a challenge to extract data. Also, consider that for each thing out there, there are likely to be a lot of copies! I am sure that a scan of all the TiVos in the country would demonstrate that lots of people are still catching up on older episodes of 24 and American Idol.

I need to refine my question a little bit, then, but I am afraid it will be difficult to track down defensible sources for it. I am more interested in knowing about the growth rate for data that can be integrated into an actionable information environment. I may not care about the bits comprising that specific episode of 24 that is sitting on millions of DVRs, but as an advertiser, I might be interested in profiling which households have watched which episodes and at what kind of time shift.

Anyone have any ideas?

1) Greater emphasis on embedding and integration of business intelligence componentry into operational and analytico-operational applications. 2006-7 showed that there was a never-slaked thirst on behalf of the mega-vendors to gobble up vendors providing the various component capabilities for end-to-end enterprise information management. As such, most of the big BI vendors are now absorbed into even larger monoliths, and the result is the further integration of their capabilities into a stacked set of offerings. Second, smaller BI tool vendors (open-source included) are demonstrating more and more OEM involvement with vertical solution vendors in the telecommunications, health care, financial, pharma industries, etc. Third, Microsoft is pulling desktop-users up the stack as more BI functionality is provided (consider the launch of Sharepoint, the inclusion of advanced analytics into Excel...). In addition, we are seeing more development of business applications that exploit BI within real-time (or near real-time) operations, such as embedded call center analytics. This points to a trend to move business intelligence out of its own arena and further its inclusion into the mainstream. My prediction: Across the board, vendor messaging will focus on performance management and integrated reporting within both "pull" and "push" frameworks, increased embedding of reporting and performance metrics into existing operational applications, and establishment of BI capabilities as a "dial-tone" service.

2) Recognition of the criticality of location: Mobile phones have it. Automobiles have it built-in. Now you can purchase one for less than $150. What is it? A device that can access and manipulate data from geosynchronous positioning satellites. Of course, I am referring to the soon-to-be ubiquitous GPS device that hangs suctioned off a growing number of windshields. Most of the more sophisticated mobile devices have it also. In fact, the major geographic data companies are in midst of acquisitions, one by mobile device manufacturer Nokia. This underscores a growing understanding of the nature of location, not just in determining how to get from here to there, but also information about all the points in between, as well as where things happen. Consider this: no matter what, almost every business activity takes place somewhere, and the more you know about how these activities reflect the location in which they take place, the better your operational decisions will be. My prediction: increased incorporation of geographic business intelligence into analytical applications and platforms.

3) Emergence of policy management solutions to supplement MDM: Many of the MDM case studies are largely siloed consolidation and management of a focused data collection. We have seen a number of customer data integration hubs, product information management systems, even geographic data hubs, largely for analytical uses. However, the value of MDM is largely increased when the master data is used in both operational and analytic environments. But to establish reasonable master data services for operational or transactional applications, the MDM systems must be able to demonstrate management of the relationships between master objects within the operational contexts, which are typically designed to address defined business process requirements. On the other hand, the combination of siloed data sets into a master environment introduces numerous data quality and business operations requirements to be imposed across the master data as well as the upstream data sources. This means that the policies guiding business operations and data quality management must be absorbed into the MDM environment and integrated into the information flows. My prediction: as the importance of policy management for enterprise information management is observed, vendors will introduce a "metadata-like" mechanism for managing collections of business rules that compose the business and information policies to which master data must comply.

Well, now that the predictions are done, I guess I can wrap it up for 2007. Happy New year to everyone, and best wishes for a great 2008!