Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

We're here, dirty data, complex business processes, inconsistent integration points - sounds like what an EDW/ADW is supposed to help solve right? Parts of it anyhow are solved by EDW/ADW, other parts must be solved by accountability of end-users, still other parts must be solved through SOI (service oriented integration, under the SOA stamp).

We've established rule #1: in a sea of data throughout our enterprises, the single most important data point is the business key - the one and only reference across the company that means something to the business, and allows the business direct access to the data set they are after.

Are we ready for rule number two? Not quite yet. Let's explore dirty data further. Not to change track, but Information Quality is extremely important. It's not just about the data itself, but it's about the people, the business processes, the metadata, and the metrics and measurement all used to ensure continuous business improvements.

Dirty data, and broken business processes can make a company "bleed money." And that's just the START! Data Models that help increase accountability from end-users, and systems architectures that help raise the visibility of business process problems help stop the bleeding, and can save millions of dollars a year if done right. But to understand these statements, we must walk through just how the systems got this way.

So we take the case of the broken business, customer SLS123, we just lost $30M to our big competitor because we took 5 weeks to respond, and our competitor took 3 weeks to respond. Please note, just because they responded quicker, doesn't necessarily mean that the quality of their product is better - it just means they stream-lined a portion of their sales, finance, and contracts communications. Now if they deliver faster, with higher quality - then they've truly got us beat, and we will go out of business if we don't do something to correct the situation (keep up).

By the way, this is what ERP systems attempt to address, and sometimes do a good job of it, but obviously they leave a little bit to be desired (due to high levels of customization), hence the usage of additional tool sets like EAI, to move the customer into CRM systems and through even more complex business processes.

After examining our business process here's what we find:
Sally takes the first contact call
Sally assigns SLS123 to the customer record
Sally pre-qualifies and fills out some basic information, to which she accidentally enters the wrong address, or uses special characters to represent information that she can't store in the source system.
But because Sally wants the bonus for this customer, and doesn't want her sales counterpart Joe to get the bonus, she uses her own special characters that only she understands and can interpret to management.
Sally then hands the account off to Finance, and sends an email to Jim, whom she also works closely with because she has a good business relationship with him.
Jim in Finance pulls up the customer record by name, an auto-synchronization routine in the source systems, moved the record from sales to finance last night and changed the account number from SLS123 to FIN456.
Jim then walks through a series of checkpoints on the application, and has to call Sally to understand her encoding of the special characters (over time, Jim begins to understand it, but doesnt annotate any of the metadata).
Jim then changes parts of the application, sends the FIN456 customer to management for approval/disapproval.
Financial Management then approves the customer FIN456, calls Jim and says - pick up the customer, it's ready.
Jim then says, good to go - marks the record for upload to Contracts.
That night the synchronization system moves the record to contracts, and promptly changes the customer number again to CONT259.

And the cycle goes on, the complexity increases, the touch-points increase. When we look at this particular scenario we discover that there are critical touch points and manual approval mechanisms that must be in place, we also discover interesting auto-synchronization mechanisms hidden in our legacy systems, or even in our re-engineering of the legacy into ERP and CRM.

We finally discover that there are unnecessary processes that the data goes through which neither improve the quality nor speed the process up. These are the business processes we wish to eliminate to stop the bleeding. Now there's the data set. One customer: John Smith, 3 Account Numbers - SLS123, FIN456, CONT259. Can the business trace John smith at an enterprise level? Not very effectively. Does the business have deep visibility into their data supply chain? No.

Business Rule #2 for effective profitability:
Once a key is assigned to a data point, it MUST NOT CHANGE.

Not in a box, not with a fox, not here nor there, not anywhere (Dr. Seuss) - the business key must stay as a consistent representation of the data point from this point forward.

Business Rule #3:
If the key changes, you can be certain that there is a break in the business process at that point, and that you are bleeding money.

Business Rule #4:
If the key changes, you can be certain that there is a flavor of ownership of data (kingdoms, fiefdoms) within your organization, and that there are parts of the organization who are guaranteed to produce different financial results - every time, and nearly on purpose (embedded in the culture of that business unit to say the other units are "wrong" in their view of the customer).

Business rule #5:
Use of abstract character annotations to mean certain things in metadata format are usually an indication that the incentive from corporate is misplaced. It also means that the business users cannot be held accountable for poor audits, nor are they incented to improve the data quality, even though the data itself is "broken", as is the capture system.

As we continue down this track, we will discuss how an integrated data store (ADW/EDW) can help pinpoint some of these problems from a metrics driven perspective - but only if the right models are in place. We will also begin showing how to help business users become more accountable in their positions - and actually begin to issue change requests, and allocate dollars to fixing the source capture systems, thus stopping the hemorrhaging of the company, while making it more nimble and stream-lined.

Thoughts? Shout out, enter your comments below... I would love to hear from you.


Posted May 27, 2005 4:57 AM
Permalink | 2 Comments |

2 Comments

Dan, your points are valid. But, the scenario does not look realistic. Also, a bit of proof-reading would not hurt.

Think about it!

Is it a $300 million contract or $30 million contract? Can you imagine placing such a big order on the phone? ( Pure pun : What about offers/counter offers? What about kickbacks?)

I have a hard time visualizing the sales scenario. May be it's just me.

Mike

Hi Mike,

$300M contract or $30M contract, Yes I suppose I overexaggerated the price. Next time I'll try to put it more in-line or make it more realistic. In this particular case there are contracts between those two levels, and it's not a single phone call that actually closes the deal, but several months of negotiations. There are quite a few POC's that must be demonstrated, and in fact the competing vendors have to ultimately work together to win parts of the awarded sum.

Maybe I should have specified that it's not so much about the amount as it is the principles involved. Certainly one would hope that a $30.00 item wouldn't take as much to manufacture as a $300.00 item. The same can be said for the difference in quality between a $300.00 and a $3000.00 item.

One of the points I want to make (that I'm not sure I was clear on) is: as price increases, complexity increases. As complexity increases, so does the need for accuracy in the manufacturing process. As the need for accuracy increases, the risk of failure or rejects increase. In other words the number of business problems that can crop up continually rise as price rises.

Being nimble at $30M is much different than being nimble at $300M. Think about what it costs to send a probe to Mars, or even a manned mission to Mars - vs sending an expidition up Mt. Everest.

All in all, great feedback - I'll try different scenarios in the near future to illustrate different issues - this one just comes from experience in US Government Contracting (I changed the prices to protect the corporations).

Thanks,
Dan L

Leave a comment

    
Search this blog
Categories ›
Archives ›
Recent Entries ›