Blog: Dan E. Linstedt Subscribe to this blog's RSS feed!

Dan Linstedt

Bill Inmon has given me this wonderful opportunity to blog on his behalf. I like to cover everything from DW2.0 to integration to data modeling, including ETL/ELT, SOA, Master Data Management, Unstructured Data, DW and BI. Currently I am working on ways to create dynamic data warehouses, push-button architectures, and automated generation of common data models. You can find me at Denver University where I participate on an academic advisory board for Masters Students in I.T. I can't wait to hear from you in the comments of my blog entries. Thank-you, and all the best; Dan Linstedt http://www.COBICC.com, danL@danLinstedt.com

About the author >

Cofounder of Genesee Academy, RapidACE, and BetterDataModel.com, Daniel Linstedt is an internationally known expert in data warehousing, business intelligence, analytics, very large data warehousing (VLDW), OLTP and performance and tuning. He has been the lead technical architect on enterprise-wide data warehouse projects and refinements for many Fortune 500 companies. Linstedt is an instructor of The Data Warehousing Institute and a featured speaker at industry events. He is a Certified DW2.0 Architect. He has worked with companies including: IBM, Informatica, Ipedo, X-Aware, Netezza, Microsoft, Oracle, Silver Creek Systems, and Teradata.  He is trained in SEI / CMMi Level 5, and is the inventor of The Matrix Methodology, and the Data Vault Data modeling architecture. He has built expert training courses, and trained hundreds of industry professionals, and is the voice of Bill Inmons' Blog on http://www.b-eye-network.com/blogs/linstedt/.

Well, it's been a thousand miles, and a million years since I've seen a good metadata interface GUI - or for that matter, a complete enterprise metadata data warehouse (MDDW). Something that not only reports and integrates the metadata but also allows modifications from a front-end user perspective. Something with security, thin client access, read-write (bi-directional ability), and so on. In this blog I discuss what I'd like to see in the future of BI and metadata management. Right now the market is very dis-jointed. This is somewhat of a one-sided rant, if the vendors would like to respond - I welcome the new information.

I've seen a lot of metadata products, and heard about a lot of metadata products. Of course I've read the reviews by Gartner and Meta, and so on... But I have yet to walk into multiple clients and see a successful tool being utilized. Companies claim they have customers, but where are these implementations? One would think that if the vendor makes a claim, that the company which produced a metadata success would be winning all kinds of awards and showing off their enterprise metadata project.

Where are we now?
Since this hasn't happened, and since it seems like all the metadata features are still detached from the real business interfaces, and since the metadata collection and integration devices are disjoint, I thought I'd discuss where the market should go. This entry is based on comments I hear at client sites, and from consulting experience.

The questions I get frequently are:
* Which "tool" is the best metadata tool on the market?
* Why can't I update and maintain my metadata from within my BI reporting tool?
* What's the difference between business and technical metadata?
* Why do I need metadata anyhow?
* How can I tie my [technical] metadata to my business requirements?
* How does business process workflow play a part in my metadata?
* How can I lock down my metadata with roles and responsibilities?
* Metadata Data Warehouse (MDDW)? What's that?
* Why do these metadata repositories cost so much but offer so little?
* Is there a best of breed metadata tool out-there?
* How do I build my common metadata repository incrementally?

Ok, you get the point.

Where's my Metadata?
Business users are screaming for it (they've been screaming for years). BI reporting vendors (it seems) have only scratched the surface, by offering simplistic "pop-up definitions" of terms, and at that - there's no ability to maintain, manage, or control multiple term over-rides depending on login and lines of business. There's no differentiation across tools to manage the metadata. There are a few great repository tools for collecting a variety of metadata sources in the background, and some of these are rather expensive. Below I've listed (short list) a few places I see metadata pop-up:

* Excel spreadsheets (from-to or source to target mapping control documents)
* ETL coded routines
* ELT (SQL Views)
* Excel Spreadsheets (aggregations and groupings of user data sets)
* Word documents (requirements and addendums to projects)
* Source and Target data models (logical and physical)
* BI Reporting tool repositories (computational fields, yet another logical model)
* EAI, EII, and web-services tools
* User hierarchies and security (in the database engine)
* Database metrics (size, growth, down-time,CPU utilization)
* Business Process Workflow engines
* Excel Charts
* OLAP Cubes
* Microsoft Access Databases
* E-Mail
* Images (charts, graphs, pictures, photos)
* XML Files and schema tags
* Organizational Charts
* Visio Diagrams
* Web Pages
* Code (of any type: .NET, VB, Perl, PHP, Java, etc...)
* Semantic definitions and ontology’s (RDF and OWL)
* Neural Nets and learning systems
* network topologies
* Security topologies

I'll huff, I'll puff, and I'll blow your metadata in!
I've yet to see a proper metadata tool that addresses more than a surface scratch into some of these areas. Don't build your metadata solution out of straw, or on a sand-bar... A good metadata project is just that: a PROJECT. It needs to be treated like a standard project to be completed properly. Bill Inmon discusses the need for Metadata in DW2.0 (as a requirement to be DW2.0 compliant). Don't think you can simply "slap metadata in place" - it just won't happen. Buying an off-the-shelf tool for it's repository is a good start, pending it's interfaces to GRAB metadata, but most of these tools don't provide the GUI interface to update metadata, nor do their repositories VERSION metadata or provide a "metadata data warehouse".

Chicken Little the Meta Sky is Falling!
No, it isn't... Metadata is typically an after-thought, but with compliance breathing down our necks, and massive consolidation projects on the rise - metadata is becoming more important. (What is a meta-sky anyhow?!?)

I say Holmes, what do you make of this?
Well, Sherlock Holmes I'm not... but my intuition tells me that EII is positioned to access all of these components, a good repository on the backend to manage it, and with "hopefully" new EII user-front ends, metadata can be managed in an ontological format.

What experiences do you have with metadata? Are there any particular vendors you've worked with that you like/dislike?

Thanks,
Dan Linstedt
We have openings in the Masters of Science program for Business Intelligence, see more about the program at: http://www.COBICC.org


Posted May 15, 2007 7:03 AM
Permalink | 2 Comments |

2 Comments

I guess you read my mail after all :-)

After some unfortunate long period of cheap metadata gurus, books, and solutions, a mature need for metadata is slowly evolving. I would say there are three major modern urgent problems require and could be solved by dedicated metadata applications only, which makes metadata more and more important:

Impact Analysis
Data Lineage
Regulatory Compliance

There is no doubt that ASG Rochade is one the best metadata engine. Of course, that hierarchical database is challenge for users expecting relational capabilities. That is major reason that we developed and integrated our Copula ETL engine version specifically for Rochade, solving all listed technical issues, http://www.dataintegrityinstitute.com/Enterprise_Metadata_System.htm .

Regarding your questions about integration of business and technical metadata, we think that only successful corporate metadata system is one which fully integrates Structural (you call them technical, perhaps), Business and Operational metadata. Of course, that integration is a metadata application by itself that has to build. Nobody could provide such solution of the shelf.

Metadata could not be solved by corporate naming convention that tells how you are going to compose table and column names. A robust corporate metadata system is must.

Drago Pejic
info@DataIntegrityInstitute.com
www.dataintegrityinstitute.com

Leave a comment

    
Search this blog
Categories ›
Archives ›
Recent Entries ›