Blog: Dan E. Linstedt

Design: Subject Oriented Versus Function Oriented

For a long time Dr Ralph Kimball has spoken about subject oriented design. Many have made a living off of producing subject based data marts. One of the problems this has lead to is a series of loosely coupled stove-piped answer sets that are then "discussed" in the light of an enterprise data warehouse. I've been teaching, talking and writing about (over the last 10 years at least) a notion called Functionally Oriented Design. In this entry I will briefly introduce my notions of these concepts.

]]> When we examine subject oriented design patterns we discover that the true nature of this design is to meet the needs of a specific group of people, furthermore - performing a very specific task or attempting to answer a highly specified business question. In fact when we do additional research around "subject-oriented design" we find a number of interesting facts:

"Feature-based
Decompose problem vertically into separate features, perhaps with a distinguished base feature containing common functionality; design and implement each of these features separately; then compose feature implementations to produce a complete system."
"Staged
Define the problem and design the system. Then decompose the design horizontally into implementation stage’s each of which extends its predecessor. Implement these stages successively, extending system through composition. "
http://www.research.ibm.com/sop/papers/position96.htm

What does this mean?
Well, if it's used as a design paradigm for an enterprise - this usually means that the enterprise should be broken in to lines of business, and then further broken in to specific feature-based activities. I can easily see where we would get our notions of facts for a specific feature set, that is: facts at a specific grain, designed to answer a very specific set of questions.

What about the staging or re-composition?
Interestingly enough this is what conformed dimensions attempt to do, along with Junk and Helper tables... Once decomposed, then the next step is to "stage" the problem - horizontally along the implementation to extend it's predecessor. This might be related to additional aggregate fact tables, and even more conformed dimensions _at a different grain_. However this usually doesn't extend itself beyond a single line of business. Which is why, when we examine Dr Kimball's enterprise Service Bus, a line has been drawn around the loosely coupled subject oriented data marts and "assigned the term" of data warehouse.

The image seen here, is one representation of many that Dr Kimball has subscribed to over the years. http://www.dwmantra.com/concept_index.html

As Bill Inmon states: "A school of minnows (swimming in close proximity) doesn't equal a whale." At the same time, the original definition of data warehousing includes "subject-oriented" in it's purpose.

Lately (for the past 15 years or so) I've been experimenting with Functionally Oriented Designs. Why? Because the business process is very important to the enterprise data warehouses that I've been building.

The nature of functional design is to couple form and function together to solve a problem. In this case, the form is the data model, and the function is the business process. By doing so, we solve different types of problems for the enterprise data warehouse. Don't get me wrong, I believe in Star Schemas for end-user access, cleansed and organized data, but I do _not_ subscribe to their use as an enterprise data warehouse. More on that later.

Functionally Oriented Design has taught me that it is possible to build a system based on business rules and to maintain it as quickly and easily as the business changes. More to the point, if we design our enterprise data warehouse properly, we can actually store the history of _some_ of the business rule changes within the structure of the enterprise warehouse as well. Thereby opening up new territory for "comparative design analytics" in review of "what have we changed business wise, and was it a good decision..."

Functionally oriented design is focused on the business process, and therefore the business keys that drive through the business processes. Because of this notion, the design actually runs horizontally (instead of vertically by line of business) across the entire enterprise. The design focuses on tracking and tracing all information by business key from the time it first enters the business to the time of delivery of the goods or services.

The Data Vault modeling paradigm is based on functionally oriented design views, we'll discuss more in the next entry. For now, if you have a different view on Subject Oriented Design I'd love to hear it.

Thanks,
Dan Linstedt

The Business of Data Vault Modeling (Book) Available

My book on the Business of Data Vault Modeling, approach and archtitecture is finally available after 7 years. If you'd like to purchase it, you can grab a copy from LULU.com (here: http://www.lulu.com/content/1371769) If you'd like a signed copy, please contact me directly with all your information. Bill Inmon has kindly written the forward for the book.

The Data Vault Model and approach to implementation is the next paradigm shift in accordance with DW2.0

]]> It has been a long time coming, but we finally have all the right pieces in place now. The Data Vault model and approach to implementing EDW's for real-time, unstructured data, operational data warehouses, and strategic data warehouses is critical to success moving forward. If you want to find out why (from a business perspective), the book is a great resource to guiding you through the mitigation strategies that exist today.

The Data Vault modeling principles are based on (backed with) SEI/CMMI Level 5, Six Sigma, Lean Initiatives, Cycle Time Reduction, and Business Activity Management components.

Let me know what you think of the book.

I've got new blog entries coming soon.

Thanks,
Dan Linstedt

Operational Data Warehousing on the way...

Before we get to Dynamic Data Warehousing, we need to first reach Operational Data Warehousing. Now I realize that I'm not the first, nor will I be the last to use or even possibly abuse this term. In fact if you search on the term today you'll get tons and tons of hits. I do however believe that Data Warehousing and BI as an industry have gotten slow, and become somewhat of a laggard in terms of keeping up with technology. Just look at the adoption curve of DW2.0... It simply isn't there yet (wish it were). Anyhow, in this blog let's take another look at the ODW as Bill Inmon and I are beginning to discuss it.

]]> First, I must say: Thank-you to Bill for not only being a great friend, but a wonderful mentor to me. I must also say, thank-you to Claudia Imhoff and Colin White for writing about Operational BI lately, and of course all my other friends out there who continually amaze me by answering my simplistic and absurd questions.

On that note, I've been pondering (and asking Bill for help) Operational Data Warehousing. I've also been blogging on the subject lately, and as of last week - had the wonderful opportunity to share my questions with my good friend Jeff Jonas (see his blog here). More on that in my thought experiments section, where I'll also be blogging on Form versus Function and some new advances in computing sciences.

So to the point: Operational Data Warehousing as it were requires:
* good form
* strong functionality
* streaming real-time data
* scalability, and flexibility

When I talk about real-time data, I'm not talking about "every 3 to 5 seconds, I get 500 transactions or so..." No, I'm talking about every 2 to 5 microseconds, the warehouse receives burst rate mini-batches of 500 to 10,000 transactions across multiple feeds... In other words, AS the transaction is created and pushed across to other source systems, so it is pushed directly in to the warehouse on an Operational basis.

In some cases, the objects doing the data collection and generating the transactions do NOT keep a copy of the transaction. In these instances, it is important to realize that the real-time fed data to the ODW IS a system of record. Now, as a DW2.0 compliant architecture, we are housing SOR data and NON-SOR data in the same structure, in the same place, at the same time. By Non-SOR data I mean: anything defined as "arriving from an operational source system which keeps transactional history" *** It has NOTHING to do with it being batch or non-batch ***

Ok, so there was a comment on my blog from a good friend: Walter Smetsers, requesting clarification of a statement about "will we continue to need operational systems" once we build an ODW... The answer today is: maybe. In the future, as new systems are built - convergence will take hold and the answer may become: no.

In an ODW, we not only have the capacity, but also the capabilities to program directly on top of the ODW the operational applications themselves. However in order to make this happen, we also need a Master Data layer inside the ODW, along with a Metadata Layer, and a Master Metadata Layer. All of this MUST be coupled together, and managed through an ontological function.

Ok - enough blathering, do we have one of these or don't we?
Yes, we've built one, but it doesn't YET have all the components it needs.

Where is it?
Secure Symbology Incorporated, it's a Data Vault modeling architecture called the Serialization Vault that we've built for the national E-Pedigree mandates from the FDA. I've heard that the WHO (world health org) is interested among other parties.

How does it work?
We accept data in real-time from collectors on the manufacturing lines to a central Data Vault modeled data warehouse. We keep the data itself separate, and through a logical model and metadata layer we can re-assemble the disparate data sets to provide the drug manufacturers a complete picture.

We also have a layer of operational systems on top of the ODW, allowing data to be logically updated by the application. I say logically because in keeping with the DW2.0, it stores history of the original transaction, and merely inserts new information rather than updating in place.

There are a few other customers who've had one of these in process for years. I'd be happy to put you in touch with them.

BACK to ODW...
Are vendors supporting these concepts today? Not directly. You can build one on any RDBMS system, or column-based database, as long as application programming logic can access the data underneath directly. Operational BI is coming to the fore-front, and there are a number of young tech vendors coming to the table to meet the challenge, but it will be a while before the market space "grows up".

Does this mean my ODW is treated just like an operational system?
YES! It is an OPERATIONAL SYSTEM that has converged with a DATA WAREHOUSE, therefore it has the same requirements as an operational system and an EDW at the same time. The best type of data modeling for this is a normalized format, for scalability, flexibility, and auditability.

I'd love to hear your opinions, thoughts and questions.

Thanks,
Dan L

Unstructured Information Pushes Dynamic Restructuring

I've just completed Bill Inmon's brand new course on Unstructured Data using his new Unstructured Data ETL tool. It's been very eye opening. Every time I meet Bill I'm always learning something new. There was a discussion at the end of the class that asks the question: WHAT do you do if you FIND "structural definition elements" in unstructured data that AREN'T represented in the EDW??

]]> So what is interesting here, is the notion that mining unstructured data gains knowledge or metadata by association that provide definitional elements _about_ the information held within. What we can do, and what we need here is truly Dynamic Data Warehousing - which includes the ability to BRIDGE structures, along with creating new structures.

To get to the unstructured data mart, we should be going throught the Dynamic Data Warehouse. Remember, I'm using the term Dynamic to include STRUCTURAL CHANGE, INDEX CHANGE, etc... So what that means is: that the structure which is discovered through unstructured data surfing is built via nerual networks, then it is subsequently graded on strength and confidence, and finally - optimized and adjusted by the neural net as new unstructured data is fed in.

This is what the Dynamic Data Warehouse looks like to me. The follow on is dynamic cubes, and once the unstructured data reaches the dynamic cubes we (I'm sure) will be surprised at what we find...

Stay tuned for more info later.

Cheers,
Dan L

Dynamic Data Models - Automorphic Changes

It seems people have taken the term "Dynamic Data Warehousing" and abused it. They've made it out to be about "Dynamic Data" and completely ignored "Dynamic Modeling", or dynamic restructuring as the case may be. Automorphic means self-changing, self-adapting. In this entry we'll talk about different capabilities of Dynamic Data Warehousing and the changes to data models as they grow.

]]> First, let's define what we mean by Dynamic Data Warehousing:

My definition of DDW has come to mean:
* Data models that can adapt to the incoming data based on A.I. rule sets, learnt data patterns, linguistics, metadata, and associativity.
* Load patterns that are driven by the changes, to move data from "point a" (the source) directly to point b (the target, or the dynamic model)
* Indexes that shift, are created and dropped on the fly based on load patterns and query patterns
* Learning systems attached to the firm-ware of the device to watch, and learn about the metadata - tying the metadata together is an important step.
* Adaptable cubes - dynamic cubes or in-memory aggregations based on "temperature data" (hot, cold, warm). In other words, in-memory ROLAP solutions that are built based on metadata and cube structures, attribution of data sets, and queries or the questions being asked.

Now this may all sound really interesting, and extremely future based - but I can assure you - it's not. I am currently working on solutions in my lab which entail the execution of portions of these elements. The hardest one (you might think) is the dynamic modeling, or dynamic restructuring of the database... well, let me tell you - nope! That’s not the hardest piece (when the Data Vault modeling architecture is used)...

Keep in mind that the DDW is one or two steps beyond the Operational Data Warehouse, which I've just begun writing about. Also remember that the term: DDW retains all the responsibilities of the "Data Warehouse", as in: time-variant, non-volatile, granular, etc... That said, the question for Dynamic Data Modeling then becomes: how do you keep history on massive volumes of information without losing value, and without "reorganizing" or altering existing structures?

The answer is to come later, but with the Data Vault modeling methodology it CAN be done...

So how about 3rd normal form?
Sorry, it seems to be incapable of handling dynamic structure change. Why? The same reason that it fails as a Data Warehousing architecture in the first place. Parent-Child relationships embedded in the tables, and then: placing the structure over time. changing the structure of a 3NF DW is bad enough, let alone trying to alter it on-the-fly during loading and maintain existing history. This requires super-human strength, massive amounts of disk (to copy the elements), and sometimes changes the MEANING of the data when the structure changes.

"Danger Will Robinson!" (quote from a U.S. T.V. show ... lost in space, from the 1960's) http://en.wikipedia.org/wiki/Danger,_Will_Robinson

Ok, so what about Star Schema's?
Well, if you read through the definitions of Star Schemas AS ENTERPRISE DATA WAREHOUSES you quickly find that it's not the right fit, hence the new Generation 2, DW2.0(tm), and other new modeling concepts like the Data Vault.

Have you ever tried to change the structure of a conformed dimension? Does it indeed get harder as the system grows, and/or the more conformity it has? Does it slow down your development efforts?

Yes to all of these (at least from my personal experience). Does that make Star Schema bad? NO! Star Schemas are AWESOME, WONDERFUL, and the ONLY solution to work for OLAP, and Drill Down... Do they have a place in the DDW? YES! ABSOLUTELY! well then, where?

They have a place as adaptable cubes. Something funny happens when Star Schemas are used as SINGULAR STARS to LOGICALLY define VIRTUAL marts. They work extremely well, and as long as they are logical (not physically implemented), then dynamic memory cubes can become a reality. That's right! IN-MEMORY CUBING, it's happening already in certain DB engines, but it's not yet dynamic.

However, as a DDW foundational structure, we need something else. The Data Vault Model seems to be (today) the only other choice available that is actually capable of executing on this dream. We'll talk more about this in my Data Vault blog on http://www.BetterDataModel.com

Cheers for now,
Dan Linstedt
DanL@DanLinstedt.com

Part 6: Secrets of the Masters

To follow on with our series, we'll dive in now and explore some of the elements needed for a repeatable, consistent, and redundant project. These are components that make the project book completely usable - without these pieces, the project methodology usually sits on a shelf and gathers dust. What we are aiming at is: the hope of reducing overhead costs, reducing errors, increasing productivity, and increasing agility of I.T.

]]> Welcome to the next entry in the series. In this entry we'll dive in to some of the physical components you want in your project binder to help make it work. Despite what some believe (that this is cumbersome, overly documented, pushed to the edge, too many standards) etc...

Once followed, it actually makes project work repeatable, consistent, and reliable. It puts you and your team in a position of continuous process improvement, and continuous success. There are a few key deliverables that absolutely must be put in to the project binder.

1) Service Level Agreements - SLA's. These are a phenomenal way to communicate acceptance of everything from default values to problems and issues that arise during project implementation. Without SLA's, the project can easily be "blamed" for not meeting goals, not building what was expected, not getting it right.

The hard part? Getting the business to realize that this doesn't mean "it's set in stone" - it simply means that they understand the level of service currently being implemented, and they agree with the output _at that time_. Once we've achieved this level of understanding, and teach the business that they can change their minds (by signing a new SLA) then we can take steps forward.

SLA's function as change requests, issues and mitigation choices, and implementation direction controls. Standard SLA's should be a part of every project, and should be carried around by the project lead in their back pocket.

They don't have to be long (usually not more than one paragraph) but they do need signature lines: Project manager, one of the sponsors, and a business technical lead.

2) Business Technical Lead. What is this? This role is usually assigned to someone on the business side of the house. Someone who then interfaces with the project manager in I.T. on a daily basis. They are business driven, but have a knack for I.T. and a deep understanding of how to work the technology. They are responsible for setting up all meetings with the business users, coordinating the rooms, writing the user help manual, generating the first cut of metadata and definitions, and being a part of the immediate "testing team" which provides results / runs reports against your production data.

They are the ones who call emergency meetings when a hard-stop is reached in the project. They also work in tandem with the business itself to ensure the project doesn't fall off-track, or off-focus. they communicate with the Project manager about pending business changes. These resources are invaluable. If you don't have one on your project, your likelihood of _long term_ success is limited.

3) Expert Knowledge. All too frequently consulting companies bring in the expert to help close the sale, I.T. departments are guilty of the SAME THING! Then, they wisk the expert away back to some production fire which is not the project at hand, never to be seen again. This doesn't work for successful projects. Successful projects _need_ the expert on a consistent basis, and if an expert isn't available locally - get one from outside. Hire one as a consultant, bring them in as a sub-contractor, etc.. Any way you can. These experts should have the following skill sets:
* Have handled large projects
* Have built (actually technically implemented hands-on) with large volumes of data (25 TB+)
* Have expertise in data mining, understanding of data patterns
* Have multi-database engine experience (not just a single database engine)
* Understand business requirements
* Have a grasp of _multiple_ ETL, and BI tool suites
* Have working knowledge of 3rd Normal Form, Data Vault Modeling, and Star Schemas
* Understand ODS, Stage, Data Warehouse, and Star Schemas
* Be able to articulate definitions of compliance, accountability and governance.

It's a plus if they've worked on real-time data warehouses, or have had government experience, and gained SEI/CMMI Level 3 experience.

4) Have a subject matter expert: a SME who's also well versed in Cobol (if there are Cobol sources to pull from). Someone who understands the business from a data and systems side. Someone who can help identify exactly what you have, and what you don't have available to source from, or target to. Someone who can identify the problems with the current source system, and share the existing change requests that the system is undergoing. This person should have daily interaction with the Operational Systems and the teams running those applications.

Ok, so in this entry I diverged a bit. I'll get back to the templates in the next entry in the series. In this entry I went back to rule #1: identify roles and responsibilities - these are key roles and responsibilities that make projects successful out of the gate. Even small projects at mom & pop shops, these roles and responsibilities are important.

Hope this helps,
Dan Linstedt
DanL@DanLinstedt.com

New Skills Required - Interactive BI

Do you still think that knowing flash and actionscript is not a newly required skill? Have you seen the latest version of Crystal Reports from BO? It now contains a front-end product that used to be called Excelsius, which is a flash-based front-end BI dashboard library. You can drag and drop buttons, charts, pre-built reports, and other things on to the different scenes in the dashboard. No more flipping pages, and writing PHP code or Java code to exercise BI on the client side. It's a highly protected environment.

]]> I'm on site this week at one of our Data Vault customers, and what do I find? A technical business lead working with Excelsius, a flash based reporting product (now owned by Business Objects), producing live dashboards for executives COMPLETE WITH METADATA. I know there are a few products out there in the space, and I'm going to research a little more.

It's here. The concept of Interactive Business Intelligence. The idea that you can interact with the BI in front of you, along with the metadata, the master data, the help, and even the database backend all through flash. When an Interactive BI platform is built it includes the following abilities:

* Help / Knowledge / Metadata for anything, any metric, any chart, any button on the dashboard
* Dynamic query forms according to login security
* Re-arrangable dashboards, zoomable components
* Master Data, reporting, and maintenance
* Import/export of Excel for users with privileges to do so.
* Live Learning/Training videos available WITHIN the dashboard
* The ability to CHANGE the metrics and automatically produce new dashboard graphs through an executive drag and drop.
* Data Capture Forms, yes - in Interactive BI, it sits on TOP of an Operational Data Warehouse, therefore - when the privileges are available, the form actually will allow Operational Data to be entered live in the Interactive BI front-end. Now lets REALLY start talking about "What-if" games in BI, this is where the true power comes. Keep in mind that operational Data Warehouses may allow what "LOOKS" like an update in the front-end, but follow strict EDW rules, and always always always INSERT the data on the back-end.

The dream of complete Interactive BI is not yet here, but the journey has begun. With tools like Excelsius on the way, there are bound to be more.

Interactive BI is NOT just reporting and viewing help, or drilling down and back up. Interactive BI includes a graphical experience, and the ability to really INTERACT and even CHANGE the data one is looking at, live, real-time. It includes the ability to re-arrange the dashboards (which can be done today in traditional technology), and to watch help videos, participate in training sessions, collaborate ON THE FLY with co-workers while working with the dashboard. Most of all, interactivity with the data set itself.

Interactive BI sits on top of an operational data warehouse (as I've previously described). Now, is everyone going to need this today? Certainly not. Only certain applications (today) lend themselves to this type of delivery mechanism, and only certain types of data warehouses really require operational data warehousing as a way of life. But I will say, that this is where the future is headed.

To the point, it's not just about delivering the right data at the right time to the right user; it's also about enriching and enhancing the user experience - making the data TRULY flexible and applicable to the business. It's about allowing the business to make the changes WHEN REQUIRED, see what the impacts are, and then either make a different decision (immediately), or return to the previous state.

We continue (as an industry) to discuss how business rules and business logic are moving towards the hands of the business user, Interactive BI puts this squarely in their laps. Interactive BI is another step in the development process, and requires I.T. to learn new skills for delivery. Interactive BI mixes and enriches the delivery of the right information with the delivery of on-time metadata, master data, video help, live audio and even management and alteration of Operational Data. Most of all, it's DYNAMIC, all of it is dynamic; while the data sets can be real-time or strategic in nature - the interface remains dynamic.

Interactive BI is the next big thing and sits on top of a DW2.0 compliant Operational Data Warehouse. It's early yet in the life-cycle and needs to grow up, but the base-level technology is available today for those that want to move to the next level of competitive edge.

Cheers,
Dan L

Part 5: Secrets of the Masters

As this series progresses, I've received some wonderful comments, thank-you to all who are replying. In this entry we'll talk about some of the additional skills that are helpful in managing and developing successful projects. We've touched on a few already, but I'm not convinced we gave enough credit to these items. Many people argue with standards, claiming they are over-burdening their development - claiming they can't get their job done with them, claiming they are too verbose or have too many pieces to work effectively. They revert to RAD, JAD, and extreme methodologies...

]]> Why don't I mention Agile?
Because Agile can be blended appropriately with standards to make a huge difference. My friend Kent Graziano teaches how Agile methods combined with the appropriate data modeling and other practices can really make a positive impact on a team and their performance.

I personally like Agile, but obviously with a few twists.

Skill sets that I recommend people to have on projects (both big and small) are as follows:
* Agile Methodology
* SEI/CMMI Level 3 from Carnegie Mellon U. (at a minimum, level 5 if you can get it)
* PMP Certification from the PMI
* CoBIT from ISACA
* Knowledge of ITIL -
* Six Sigma
* Lean Initiatives (from MIT)
* Function Points - Link 1, and Link 2

You might say, GEE WHIZ How am I ever going to know all that?
Well, if you know what these things are, and can find people who are experts in specific areas, then your project will be better off. I don't claim to know all of each one of these, I only just have scratched the surface on each one enough to understand how to use certain tools within each of them to achieve my project goals.

I use a blended approach, the final touch is applying the Agile principles to all the knowledge to scope the project down and get it done. Remember, I'm not JUST talking about big projects here, I'm also talking about small projects. Small projects and small organizations typically are the worst offenders, often setting a precedent that quickly breaks when the corporation gets in to rapid growth mode.

Each of these base level knowledge areas (if you can call them that) have valuable information to take away. The methodology that you build should be put in to a cell - like grid. Take for instance the Zachman Framework, pick one or two cells within to implement, then expand it out and build a Linstedt Grid (just kidding), build a simple grid underneath the single Zachman cell, and figure out where each of these types of concepts apply.

I for one believe that at least one piece of each of these concepts can be applied effectively to create the optimal EDW/BI project methodology. Something light-weight enough to be used effectively (and not sit in a corner and gather dust), yet something practical.

The key is:
* ensure that all components chosen are practical
* ensure that the components are TEMPLATE driven, so that the PM AND the team only have to grab ONE template (don't give them more than one), and fill it in - and voila, they are compliant
* ensure that the templates don't require tomes of documentation to understand. In other words, fill the templates in with pre-cursory language that explains WHAT is in each section and WHY it is there.
* ensure that there is a SINGLE risk document in the binder that explains the RISK of using only one component, and not another.. In other words - when the customer demands that processes be short-cutted, and standards be broken, then be able to explain the risk of doing so - then have them sign an SLA that states they've been made aware of the risk.
* force yourself to learn, and follow the standards/cell based methodology you just built. If you don't follow it, no-one-will!

It's not SEI/CMMI Level 5 by itself that could help you be successful, nor is it knowing and understanding what documents are available in the ITIL, it's being able to leverage the right document and process at the right time where it makes sense within the project that really drives the point home. Believe me, I've been through this more than once - I've shot myself in the foot by breaking my own recommendations, but we've been able to recover from the errors by reverting to the standards.

Also keep in mind the following idea: If you were a ship captain, and you needed to go from NY port to Egypt to pick up spices, you probably would first: pick an acceptable route according to international trade law, then you would map the route out so you knew what all the coordinates are, as the trip neared, you probably would look at the reported weather patterns and check for re-routing instructions.

This would be a constant (agile) process of adjustment DURING sailing, as weather is unpredictable. If your ship got off-course, you'd want to put it back on-course or risk: loosing your captains license, loosing your ship to rocks on some coast, or hitting a sand-bar, running aground, loosing your cargo (if it's food, it might expire). Now in these days of GPS it's quite a bit easier, especially since it's integrated with the NAV systems, but that doesn't justify the Captain breaking international shipping regulations....

Ok, you get the point. A process, which has standards and rules to follow - can be accomplished successfully with a high degree of accuracy - has a little bit of real-time leeway to accommodate shifting weather patterns, and includes some uncertainty (like monster waves which take out entire ships). Also keep in mind that to turn these huge ships can take a tug boat (or several) within the harbor, or can take a day or two out at sea.

Just like a project, when it's off course or the standards have been broken, can take a "smoke-jumping team of experts" or can take several weeks to put back on track. It can be done, it just takes time.

Now, put it in perspective. Am I saying that even projects that are spec'd to take 3 weeks can use these principles? Yes, I am. Am I saying that they require ALL the documentation that a 6 month project requires? No I am not, this is where re-using the "templatized" project methodology components comes in to play. This is where Agile takes over the plans and implementation status.

Finally never forget scope. Always apply scope to the project to make it work properly. As far as the 30,000 page tomes that SEI/CMMI offer - strip them down, apply the components to your project templates that are necessary (I've mentioned a few in this series already like versioning, and SLA's etc...) What you will find is this: building an EDW/BI solution is more like building software than you think.... there are hundreds of thousands of parallels between the two worlds.

Next time, we'll dive in to some of the components from each of these that I'd suggest you start with. Anyone care to comment on pieces they've used successfully or pieces that have been applied and resulted in a negative impact on the project?

Thanks,
Daniel Linstedt
DanL@DanLinstedt.com

New skills required for BI and Data Presentation going forward

So you've all seen Flash production movies? You've all heard of Pod-casts? How about web-interactivity without "changing pages"? I'm sure you've seen Flash produced web-sites, or played an animated game lately. This post is about the new skill sets needed by BI vendors, and Business Intelligence Analysts to survive the new upcoming wave. Those of you producing PowerPoints, or sitting in the background coding "BI Reports..." you've got a few things to learn.

]]> Alrighty then, I'm a little brazen in this entry, but I'm also excited. I think there's a new wave (ok, so it's not new) but it's hot, of where the industry is heading. I think AJAX is really cool, the web technology is awesome, but there are new delivery technologies on the rise that are starting to force us to re-think the way we interact with the business community.

If you've not learned Flash yet, you're already behind the 8-ball. Flash combined with ActionScript has been around for a long time, but AS3 and Flash CS3 creator have evolved. They are some of the hotest products out there. Give up on your old "BI 2-D non-interactive tabular reporting". You've got new skills to learn!

These days it's about interaction with the user, animation, graphical displays, and delivering learning content. Flash gives you all that, Unfortunately there aren't any "really good ActionScript libraries" developed strictly for Business Intelligence (yet). But I'm seeing and feeling like the existing BI vendors would do well here - to animate their reports, combine the static reports with interaction against the database, along with instructional video and quizzes (like do you know what this metadata means?)...

These are important skills for BI delivery folks to pick up. I am going to start delivering a Flash recorded video blog as an experiment (I know, these already exist - but mostly these are today geared towards the graphic artists).

I've got news for you, new skills are needed on the horizon. Flash and ActionScript can deliver a power-packed presentation, which can be tailored (live) according to the audience level of participation.

ActionScript lets me hook to the database, communicate with a server in XML & sockets, and control delivery of streaming content. Why wouldn't I want my BI solution to involve interaction? It seems like (to me now) that 2D static reporting is a thing of the past, although printing these components on hard-copy still is a good idea.

Flash makes drill-down, drill-across easy, as does presentation in 3D with animtaion steps.

Ok, so why am I excited? I've been teaching for over 10 years, dry powerpoint presentations, with PPT animation... Ok - a still a good way to deliver content, but take the PowerPoints, run it through Captivate, put it in Flash, then add my own dynamic actions to it? Quizzes, and the like... way cool. Now as an instructor I can tailor the instruction to the student WHILE they are experiencing the course. Talk about interactive "business intelligence" - I finally have the opportunity to use "LIVE BI" to interact at run-time with the student or business executive to deliver the NEXT STEP of the class according to what they need.

Really cool stuff. What would happen if someone like Cognos, Microstrategy, or Hyperion would put there output into a Flash movie? So you could "run through reports, add explanation, and decision pathing, and metadata + voice" then deliver the content to the business? I think interest and interaction would increase, especially if the recorded movie automatically interacted with the database to bring back live data... Talk about visualization capabilities... WAY COOL.

Ok - Dan, you're off your rocker...
Am I? Really? Haven't you always wanted to make BI interesting, and interactive? Wouldn't it be more fun? I've discussed convergence in the past, I've discussed data visualization, and interactive graphics - SO WHY NOT? Why should I code in SQL alone? Why shouldn't I code in ActionScript + SQL + XML and use the power of PowerPoints, Pathing, Quizzes, and Flash to make presentations that really pop?

My experimental blog will replace the Wiki on BetterDataModel.com - we'll see where this goes... Love to have your comments and thoughts.

A couple of products you really have to have (no I'm not a re-seller)
$40 - ZLASH, ActionScript editor IN PLACE in SWF files (movie files)
Adobe Flash CS3 Pro, $399?? Something like this
Adobe Captivate
Microsoft PowerPoint

Cheers,
Dan Linstedt

The new evolution of Data Modeling

Bill Inmon and I sat down the other day to discuss a system that we are building. We didn't have a good "name" for it, but what it amounts to is: Operational Data Warehousing. If you can believe it, what we've done is taken the Operational specifics of systems capturing data - and placed it on top of the Data Warehouse as a single integrated historical and operational data store. We are currently using the Data Vault model for this componentry. Some folks have called this "Active Data Warehousing" in the past, but we feel that this is one step beyond, in that it actually IS the operational store at the same time as being the Data Warehouse. Convergence has arrived...

]]> I've blogged about convergence in the past, it's no secret that the world is converging, and I.T. is no different. It is also no secret that EDW technology is converging with operational technology. Well, if we look behind us (20/20 is always best) we can see the divergence path of data warehousing and operational systems, and the re-convergence of these systems. Active Data Warehousing coupled with SOA, and real-time alerts coming back from the ADW have begun to turn the tables.

We have closed the gap on this one. Using the principles of the Data Vault modeling (http://www.DanLinstedt.com) we've constructed an Operational Data Warehouse (right now, Bill and I do not have a better term for this, Bill also thought that this is a new approach).

What does Operational Data Warehouse do?
One way to describe it is as an Operational Data Store with history.

Another way to describe it is: as a data warehouse with operational (raw) data.

Why do it this way?
Well for one, it provides traceability in all the data. Bringing in the RAW operational data over a web-service (as generated by the upstream machines), provides us with accountability, auditability and pure traceability. By utilizing the notions of the HUB entity within the Data Vault structures, we achieve horizontal integration across the data sets. This operational data warehouse is front-ended by web-services, and has direct integration in to the business processes. It is not fed with any sort of "batch" system, it is however pre-loaded with master data.

The structures of the Data Vault have been setup within the databases in such a way as to allow tremendous scalability and flexibility. We have physically partitioned the machines for security purposes, and scalability purposes. We can join 800M rows to 300M rows to 100M rows, and bring back 10k rows in under 10 seconds when we know what we're looking for. This setup is housed on SQLServer2005 on Windows 2003 r2, with 32 bit, 2 dual core CPU's at 2.8 GHZ, 2GB RAM.

So what's this got to do with Operational Data Warehousing?
Plenty. Operational data warehouses (a very lose term today) consist of the following requirements:
* Must be accountable
* Must be auditable
* Must be a system-of-record
* Must interact with other operational systems
* Must house operational data
* Must house historical data
* Must NOT separate operational data from historical data in the data store.
* Must be the SOURCE for a major business function
* Must be real-time (can have batch feeds, but must be real-time in data streams)
* Must be part of the business process flows.

So what are the technical requirements?
* Must be scalable
* Must be flexible
* Must NOT break history when the business changes/data models change
* Must NOT break existing data feeds when the model changes
* Must be FAST access, fast insert, etc...

And of course it MUST follow the DW2.0 requirements:
* Must have historical data
* Must not be "updated" directly (would break auditability)
* Must maintain cross-functional relationships
* Must be GRANULAR (to the absolute lowest level of grain available)
* Must provide strategic and tactical value
* Must include indexes/pointers/links to unstructured information

So what? How do I get there?
We've used the Data Vault data modeling to get there. It meets all these needs and has been blessed by Bill Inmon as the "optimal choice for DW2.0" data modeling. Because of the structures, along with the foundational approaches to loading the Data Vault, and what the data in the Data Vault represent - we've been able to construct the system described above. In fact, we have two of these up and running. One in our facilities in Denver, and one in Washington DC.

So you mean to say there "is no operational system"?
There is partially, there are many "machines" that collect the information operationally, and pass it back to our Operational Data Warehouse (Data Vault), but - they do not house the information after they've released it to us. The ODW Data Vault actually stores all the operational information from around the country, and soon - around the world.

Next time we'll dive in a little deeper as to what it means to construct one of these, and how they work.

You might already have one of these, if you do - I'd love to hear about it. As always, thoughts, comments, corrections, are welcome.

Cheers,
Dan Linstedt

Part 4: Secrets of the Masters

In this entry I'm going to get on a small tangent about "contracting" with companies that execute in a consulting realm, what to watch for, what to ask about, how to negotiate with these companies. These companies are famous for "squeezing" you as a customer to pin you down on deliverables (this I see as completely fair) in order for them to get paid, they _must_ have a set of clearly defined deliverables and timelines signed off on. However, these companies are also interesting in another light. I'll tell you a true story (without names) about a review of certain companies who pitched to solve a problem for a very large customer. Our team was involved in "reviewing" their bids.

]]> The scenario:
Customer "ABA" had a project scoped out: to build a warehouse of information in one years time frame with real-time feeds from a multitude of systems that included HR, web-capture, customer base, click-stream analytics, data mining operations, and a few other components. They had specified the first phase around these pieces (scoped down quite a bit), and then opened it up for bid. Somehow (although I'm not sure how) the "consulting companies" involved in the bid (all 3 of them) managed to find out: how much time the customer had to implement, and how much money (total budget) for the first phase.

The customer had told us: 6 months and $1M dollars (the numbers and time-line have been changed to protect the innocent) was what they had hoped that this would take. The customer brought us in to evaluate these pitches from these vendors.

So, to move on:
We evaluated the pitches, looked at all the power-points, documentation, and then called the companies making the pitch - discussed it all with them. At the end of the day (strange as it may seem) the results were as follows:

Company "A" pitched: 6 months, $1+ million (without expenses), they had pitched an elaborate scheme to get the company "integrated", but this first six months, they did NOT pitch any sort of relevant BI, reporting, analytics. This was just to get the database installed, up & working, initial feeds in place. They claimed 25+ people were needed from a variety of backgrounds (some of which didn't make sense).

Turns out, all three companies pitched EXACTLY to the time frame, and EXACTLY to the TOTAL dollar figures "available" for the implementation.

Ok, you get the point. So why is this case study in this blog anyhow? What's the relevance? I'm glad you asked...

The relevance is: none of the 3 companies pitches included (you guessed it), a) the artifacts that would make the project go b) knowledge transfer on the artifacts c) training and knowledge transfer on the systems they were going to put in place d) NO involvement from the customer employee's was specified.

And it went on from there. In other words, none of the pitches included words about scope control, delivery processes, methodologies and knowledge used to build the systems, training, mentoring. And of course, none of the pitches included _any_ of the actual employees of the existing corporation, not even the business users.

So, at the end of the project (and yes, the consulting companies pitched this too) "there would have to be *required*, 3 more "phases" rounding out the next two years, topping $13M to $15M in revenue streams, piling over 250 _different_ resources (rolling in on the job, out of the job and being shuffled around to different customers depending on resource needs) in order to complete what the business needed done inside of 6 months in order to stay competitive" At the end of the project, there was NO knowledge transfer, NO training, NO mentoring by the people, on the processes and methodologies used to get there, nor on the systems that were put in place.

The consulting companies "claimed" it would be easier for the customer this way, to not get involved. That it would be a "black box slam dunk" proposition that the customer didn't need to care or worry about what or who they rolled in/out of the project, nor that the customer doesn't need to care HOW they build the solution, just that they get it done in time and "within the budget they specified."

This is a serious serious mess. Now I'm no dummy (at least I don't think so)... I've worked as a CUSTOMER Project Manager / employee, and as a consultant. I've seen it (probably guilty of doing these things too in the past, and for that reason I'm writing these entries).

I no longer believe that this is the right way to do business. My full value proposition as an expert/consultant is really, truly as an educator/advisor/counselor to the customer - YOU! It is my responsibility to not only help you get the project done, guide it's build out, put in place the people, processes, and knowledge needed to get it done, but ALSO to train, educate, and share with the employees (both business and IT) on HOW we get things done, WHY we do things the way we do, and WHAT the methodologies are that make it work.

It is my civic duty to understand how to OPTIMIZE the methodologies, and to train you on how to optimize the methodologies that make I.T. more nimble, more successful, and repeatably over time - faster to the delivery track for lower cost. THIS is the way I operate.

These methodologies are not rocket science, they are however quite special from a DW/BI perspective, as we do have _some_ unique value propositions to overcome. But as a customer you need to ask for:

* Copies of the methodology used to build your systems
* Maps that tie the implementation roadmap to the features to the methodology and scope
* Involvement from your employees, how they "help", when they "help".
* A training/mentoring schedule, at what points in the project will the employees become TRAINED to be efficient in the methodology used to construct the system that is critical to your business. (SOME consulting companies hate this, because it "appears" as a loss of future revenue from their perspective, be wary of companies that won't work with you on this).

My whole point is: UNDERSTAND not only what you are "paying for" in services, but also what the end-product will be once delivered. Without that level of understanding, you (as a customer) will be beholden to the consulting company to come in and maintain it... there's the kicker my friends... yup - beholden to the consulting company. At the end of the project, if it's done well, you shouldn't NEED the consulting company anymore except for possibly additional training, or "new projects" where you are resource stretched.

Just be careful my friends. Find the RIGHT consulting company/consultant that meets your needs and will train you on the deliverables, and HOW to use the deliverables effectively in the future.

In my next entry we'll get back on track and talk more about projects, the type of knowledge that needs to be transferred, and the kind of training required to make these things a success. As always, feel free to contact me directly or respond to this blog entry. What have been your experiences?

Cheers,
Dan L
DanL@GeneseeAcademy.com

Part 3: Secrets of the Masters

Every good BI/EDW solution is backed by a good architecture, DW2.0 is no different. The frame-work that DW2.0 provides is a sound framework with all the components necessary. That said, in addition to the framework, architectures need to exist at different levels, as do standards, and templates. A solid enterprise data warehouse project usually contains many of the following components that implementers and consultants use to make a project successful.

]]> The components include:
1. Common Project Plan (usually around 1400 to 2200 line items)
2. Business Requirements Document
3. Technical Requirements Document
4. Work Breakdown Structure (WBS)
5. Data Breakdown Structure (DBS)
6. Organizational Breakdown Structure (OBS)
7. Process Breakdown Structure (PBS)
8. Skills Assessment Matrix
9. Risk Analysis Document (high level)
10. Risk Analysis documents for critical tasks
11. Critical Process Flow Definitions (as-is), both business and technical
12. As-is data model documents
13. To-Be (proposed) E-R diagrams for staging, EDW, and at least the first mart structure
14. To-Be (proposed) interface structures
15. Cross-Reference (from/to) table.element to table.element definition (source to target design spec with transformations)
16. Standards Templates for loading CLASSIFICATIONS of table structures.
17. Reporting Templates for providing data - usually part of the business requirements document, that is: existing reports they may be receiving. Sometimes the To-Be reports are defined in the technical requirements document. However, reporting templates should be designed according to the GRAIN of data they provide, along with the OBS that they serve.

This is not a comprehensive list, but it has the major components that are necessary in all projects. Keep in mind that a lot of these items can usually be purchased as templates from ITIL (the IT Information Library), or ISACA. Some of these things are available for purchase from SOME consulting companies.

I, being of sound mind (although some would beg to differ) hereby offer up (within this series) documents that might assist you along the way. I'll post these as we proceed on my companies' web site. Ok - enough of this... back to the topic at hand.

What are the key components?
1. It is vital, absolutely vital to put all of these (and other project artifacts) under a single, company wide version control.
2. ALL EMPLOYEES DOWN TO THE LAST BUSINESS WORKER SHOULD HAVE ACCESS TO THE ENTIRE STACK OF DOCUMENTS AT ANY TIME, that is of course unless your running a cleared/classified project. All these documents should be available for read-only access on the web, by any member of the business or technical side of the house.
3. ALL documents require sign-off sheets for changes, and adjustments - these sheets are signed and dated by the money holder, project sponsor, technical project lead, and project manager. If you have a PMO office, they should handle SOME of this documentation, it is unlikely they handle it all unless they are SEI / CMMI Level 5 sanctioned.
4. ALL documents must contain technical numbering. It starts with the Business Requirements, followed by the tech requirements, followed by the project plan.

for instance: 2.1.5.1 in my Busn. Req. doc, should match 2.1.5.1 in my Tech requirements doc (where we discuss the implementation aspects), which should match 2.1.5.1 in my project plan - the resource load, progress tracking, and dependencies to getting it done. The 2.1.5.1 title in the project plan should be a tech requirements title, major numbers like 2.0 should be a business requirements doc paragraph title. MIX the metadata!!!

Oh it get's better....
2.1.5.1 should also match the WBS - telling me what work needs to be done to accomplish the tasks identified.
2.1.5.1 should also match the OBS - telling me which roles & responsibilities (which are usually loaded in to project plans) are needed, and what the escalation paths are in the organization should something go wrong. It also attaches the sign-off document to individual components.
2.1.5.1 should also match the PBS - which define the current AS-IS processes that are involved in currently getting the work done, which of course also define the critical path elements (business process management/re-engineering) so that the TO-BE process designs are more efficient.
2.1.5.1 should also match the DBS - which defines what data is needed to meet the requirements.

In other words, the project line-item number should be pervasively applied to every document in the stack, so that monitoring project progression and completeness can be done. You can't reach Level 5 CMMI without monitoring, and automating (where possible), and you can't fix what's broken unless you can plainly see the hold-ups, delays, and large issues in the way.

WHAT??? There's no possible way I can get all of this for the whole project before I start... that would be MASSIVE effort!!
Yes and no.... it's all about scope. Make absolutely certain that you scope down what you are implementing, and scope down the documents as needed. The only documents which usually don't scope down are the project plan, and the business requirements documents. Once you have these, scoping is much easier.

Remember: that the size of your implementation team can bear a huge impact on the scope you can accomplish and the time you can accomplish that scope in. In other words, it can contribute to overall project RISK to not have enough resources. It needs to be identified in order to discuss with the business before moving forward.

Now, that said: I speak from experience, as a technical project manager in a HUGE organization, I once went from zero to all documents in place, and the project phased and then scoped within 3 months. Also within that 3 months my technical team of 3 - 4 people went from zero to delivery of a single report/single star schema. It is possible. Sometimes this is "sold" as a 90-day time box deliverable, but be-wary of the deliverables that do not include the methodology, the approach, and are missing the documents I specified above.

Why? because when the consulting company finishes the work, and turn over the efforts, if you don't receive the methodology behind it, and the documents aren't numbered, then it is VERY hard for you (the customer) to learn about what you have, much less maintain it going forward.

KEY POINT ABOUT SCOPE:
Usually business requirements documents are huge, in order to meet all the items within the BPR, it is absolutely vital to divide it in to phases (I like a 3 phased approach). This way, eventually all the requirements can be met, and in the first phase some low-hanging fruit (like necessary infrastructure) can be built along side an expandable and flexible architecture also along side NECESSARY requirements that are on-fire.

LET THE USERS WORK OUT WHAT'S HIGH PRIORITY FOR PHASE 1, once they understand they _will_ eventually get all requirements, just not in the first phase, they usually settle in to the task at hand of assisting to define scope WITH YOU. Ensure that you help or assist them to define scope of the project as you move forward.

As usual, I'm interested in your thoughts and feedback. What has been your experiences good or bad within projects?

Thanks,
Dan Linstedt

Part 2: Secrets of the Masters

With these posts I hope to shed some light on what makes projects work, no matter the scale, no matter the time-line; always with an eye on costs, overhead, and a watch on the number of errors. In part 1 of this series I introduced top-level concepts required by nearly all "good" projects. In this entry I will dive a little deeper into what these concepts bring to the table, and also add some lower-level concepts that are also necessary in successful projects. I'm specifically targeting 2nd generation warehouses, and DW2.0 - in an effort to move forward.

]]> In the last post we discussed the following ideas:

1. Singularity
2. Repeatability and Standardization
3. Measurable
4. Automatable

Lest we forget it's always about:
1. People - AKA: Organizational Breakdown Structure (OBS)
2. Process - Process Breakdown Structure (PBS)
3. Data - Data Breakdown Structure (DBS)
4. Work - Work Breakdown Structure (WBS)

We need to apply the concepts:
1. Accountability
2. Version Control
3. Governance

To everything we build, be it a warehouse, a staging table, or a single star schema. Without these fundamental tenants, we have a difficult time reaching the top 4 goals listed above (defined in part 1).

What does this mean in real terms?
a. Everything must be numbered, and versioned. Use technical numbering patterns for extensibility and hierarchical concept organization
b. Glossaries must be developed and maintained, common terminology is extremely important.
c. Common check/in and check/out procedures must be applied to ALL project documentation, no longer is it acceptable to keep a spreadsheet, project plan, or requirements document out of site, or on a single persons' PC/desktop. It needs to be available in a common shared repository by the entire team.
d. AS-IS diagrams, documents, statements, use-cases are just as important as TO-BE defining where they need to go. Gap Analysis is a critical part of the success to any project. Without Gap Analysis, it becomes difficult to MEASURE the LOE (level of effort) to get from point A (as-is) to point B (to-be).
e. ALL BREAK-DOWN STRUCTURES SHOULD BE NUMBERED! And attached to line-items in the project plan, as I've blogged about in the past, and will bring back again shortly.

If you can, or if you remember how to: apply function points. The secret number is the number of person-hours per function point. Usually for the average "already trained" I.T. employee, it's roughly 1.5 person-hours per function point. Divide the function points in to groups: easy, moderate, difficult. However, don't change the hours per function point. Why? the resource expertise leveling will do this for you.

Divide the resources into "expert, knowledgeable, and beginner". Assign risk categories to each level of resource. Then assign a multiplier to each resource group, the lower the group knowledge, the higher the risk, also - the more person-hours per function point it will take. Ideally you want each person on the project resource weighted. Now, keep in mind that these are NOT general categories for resources. Each resource may be expert in a different category. Where do you get the categories from? The Work Breakdown Structure.

Ok, so you're beginning to get the point. You need a matrix to cross-reference all this information, yes.. Start with a solid project plan, where the numbers in the project plan are numbered down to EACH element and EACH paragraph, or better yet - specific sentences in the requirements document that tie a project plan line-item to the requirements. THEN cross the project plan numbers to a work breakdown structure, by using this strategy, you can "resource load" your project plan with roles and responsibilities (held in the WBS).

Once you assign resources, you can setup risk leveling for each line-item in the project plan, and tie them to difficulty counts in the function point analysis (task overview). But I get ahead of myself.

Proceed at once to the AS-IS documentation:
a) AS-IS data models
b) AS-IS process flows
c) AS-IS resource allocations
d) AS-IS results (that the business is receiving)

Then, proceed to define the TO-BE:
a) TO-BE data models (tie these back to the results, by project plan by number)
b) TO-BE process flows (tie these back to the results, by project plan by number)
c) TO-BE resource allocations (if the team assignments are to change)
d) TO-BE results (this ties directly BACK to the requirements document and project plan by number)

Then, assess the Gap between the two, and generate a risk assessment complete with analysis and documentation, necessary people, roles, knowledge, and data. Then, assign importance to each item in the risk assessment, and tie the results of the risk assessment back to the project plan, again by number.

What we are doing is crossing the PBS, DBS, WBS, and OBS together with the gap analysis, requirements document, and project plan. So we can clearly see _exactly_ what it takes to get from A to B in the shortest time possible. We are also clearly illuminating any obstacles in the way, be it training, to hardware, to software to make it happen. We can then MEASURE our progress as we begin down the project plan.

Now, we are taking charge and governing our build process for EDW/DW2.0 - once we begin this effort, the business will begin to take us seriously, and see that we too are accountable for our delivery efforts - and every step of the way they can monitor the progress.

Thoughts? What have you experienced? Have you executed function points? (yes they're from the 1970's but they still work when done properly).

Thanks,
Dan Linstedt

Secrets of the Masters - Part 1

It's been a while since my last blog entry, my apologies. I've been heads down building companies lately, and they seem to be starting to gain steam. All that aside however, I've been thinking quite a lot over the years about these notions we hear about: top 10 this, top 10 that... most of these are about "mistakes" we can make, followed by short sound bytes about what or how to look for answers. There are books and books filled with great information about building, maintaining and deploying an Enterprise Data Warehouse, and then there are architectural discussions from John Zachman, Bill Inmon, and others. All good information....

]]> But what I've been thinking about is a practical guide, a good sense guide to how to get from point A to point B, and then do it over again, repeatably, reliably, and consistently. Not just to do it the first time, but also for those of you getting in to 2nd Generation Warehousing, or DW2.0, what does it mean? How should it be approached?

Please keep in mind, I'm no expert, and I hope I never claim to be, that title is reserved for people like Colin White, Bill Inmon, Claudia Imhoff, Ralph Kimball, and a few others who really are the true thought leaders. No, what I want to share is a culmination of 15+ years in the industry of building enterprise data warehouses and business intelligence systems. What to do, what to look for, how to do it, how to make it work.

I'm not (hopefully) going to dive in to the nitty gritty of it all (I've done that in postings in the past), but I'm going to stick to the architectural roots, and try to make business sense out of all of this.

For starters, we (most of us anyhow) are familiar with Dr Kimball’s approach to enterprise data warehousing; and when we reference his books in the LifeCycle Toolkit, we talk about things like Star Schemas, Enterprise Service Busses, and Staging Areas. But let me back up...

There are a few things that the CIO should be discussing and contemplating when building ANY enterprise based systems:
1. Singularity - cost of management, maintenance, organization, and flow - everything should have a center or master point of contact. This includes data, people, process, organization, and so on. Business have singularity (usually) in their organizational structures/reporting structures, why should data, and business process be any different?
2. Repeatability, Standardization - Once a success has been established, it should be repeated again, for a lower cost, higher value, and measured/monitored effort. CIO's need to focus on things like ITIL, CoBIT, CMMI, PMP, Six Sigma, and if a CIO doesn't understand these terms, or can't bring them to bear within their own organization, they should either be sent to school to learn these tools, or be let go from their position. The effect of deploying these measures within a corporation is to enable the corporation to grow, safely and collectively with practices that scale. The business gets better and better at singularity and success. Small shops should focus on leveraging these ideas at a high level, getting the corporate culture used to operating this way.
3. Measurable - Without standardization of people, processes, and data, measuring and estimating the next big project could turn out to be a disaster. Measuring the process in the Estimation cycle, or in the projection to completion cycle, or in the current effort cycle (level of effort aka LOE) is extremely important. Understanding the measurements of the people AND the data AND the processes involved go a long way to controlling project cost overruns, and budgetary concerns. They also help to point out where things go wrong, have gone wrong or are in danger of going wrong in the near future, thus enabling the business to navigate around the issue or solve the issue, which ever they choose to do. If you can't measure the effect of the people, the systems, processes or the data, then you won't be able to improve what you are doing, and thus make a repeatable success next time.
4. Automatable - once the CIO has gotten a handle on the processes, and the corporate culture has turned the corner (gotten used to standardized practices, and version control, document control, PMP and six sigma practices), then the CIO should look at automating parts of the business.

Why am I rehashing old ideas here that we have known for I.T. in general for years? Well, guess what: the EDW teams over the past 10 years (for the most part) have claimed: no standards to follow, it's a different project, every single customer is different, all pieces are special, new, and are exceptions to the rule... You name it, the EDW projects have done it (in a blaze of glory no doubt) in order to deliver on time.

Ok, so in walk methodologies that the Big 6 (now final four if you can call them that) from the various consulting companies. I'm guilty of that too - I wrote one of these behemoths in my past life... They are massive in documentation, enforce rigid standards, and usually are waived in the clients face as proof that the company knows what they are doing.

This isn't the problem, no the problem isn't that these methodologies exist, nor is it a problem that they are followed (which in most cases they are...) The problem is: when the consulting companies leave your realm... what do they leave you, the customer with?

Empty arms, lack of knowledge, minimal training... They turn over the project, walk in to do their song and dance, show what they've provided - but no one on your team understands the infrastructure, the standards, the closely guarded methodologies they used to build the huge solution set. It's like walking in to the empire state building in NY and not knowing where the elevators are because you can't see them; yet you know they are there.

CIO's with all the off-shoring, in-shoring, on-shoring, near-shoring, and beach belly wagging (just kidding)... have gotten lost in the fray here. Companies need to get back to doing business the way they do business, and they need to get back to contracting consulting companies that do business they way THEY need it done. Which means internal controls, internal standards, internal documentation, internal improvement processes...

But I digress. Back to the point. Enterprise Data Warehouses need guidelines, standards, processes, architectures, and workable solutions - otherwise you'll be stuck in a quagmire of mud for a bit, while stretching to reach the DW2.0 bank of solid ground.

These entries will start at high levels, and dive down a little bit to discuss how, and what we do to build successful systems. Remember, these entries come from my own personal experiences of working in the trenches - I'm open to comments on all forthcoming entries to see what other knowledge we can share.

Cheers,
Dan L

Automorphic Data Models

I've recently begun research in to this area, and am calling this "Automorphic data models" rather than dynamic data warehousing, because I think the concept lends itself better to this kind of term. Dynamic Data Warehousing seems to be an overly-used slightly abused term in the industry, and raises quite a few questions as to how, and what it is. Vendors are also using this term to mean different things. We'll let the business and the vendors work out their definition of this term over the next few years. I'm going to write exclusively (for a while - in this section) on Automorphic Data Modeling. These entries are aimed at the researches and the scientific people in the audience.

]]> First, I must apologize to all those who _really_ know this stuff. I am an architect, and an Information modeler at heart. I believe these connections exist to the Data Model architecture I wrote up called the Data Vault Model, because it is based in spatial-temporal mathematics, and because it is based on the "poor mans definition of how the brain MIGHT store/use/retrieve information." Based on these hypothesis, I can see where the mathematics of these types apply to the model. I'd love to hear from those of you as to why these theories will or won't work, it will be interesting to see how this progresses.

If we start with Websters definition of Automorphic we end up with the following:

Patterned after one's self.

The conception which any one frames of another's mind is more or less after the pattern of his own mind, -- is automorphic. --H. Spenser.
http://dictionary.reference.com/browse/automorphic

However, I prefer the mathematical definition of Automorphism:

In mathematics, an automorphism is an isomorphism from a mathematical object to itself. It is, in some sense, symmetry of the object, and a way of mapping the object to itself while preserving all of its structure. The set of all automorphisms of an object forms a group, called the automorphism group. It is, loosely speaking, the symmetry group of the object. http://encyclopedia.thefreedictionary.com/automorphism

Automorphic Groups: (Which is what I'd suggest the Data Vault model is built from)

In mathematics, the general notion of automorphic form is the extension to analytic functions, perhaps of several complex variables, of the theory of modular forms. It is in terms of a Lie group , to generalize the groups SL2(R) or PSL2 (R) of modular forms, and a discrete group , to generalize the modular group, or one of its congruence subgroups. The formulation requires the general notion of factor of automorphy for , which is a type of 1-cocycle in the language of group cohomology. The values of may be complex numbers, or in fact complex square matrices, corresponding to the possibility of vector-valued automorphic forms. The cocycle condition imposed on the factor of automorphy is something that can be routinely checked, when is derived from a Jacobian matrix, by means of the chain rule. http://encyclopedia.thefreedictionary.com/Automorphic+form

Essentially what we are doing within the Data Vault data model is a form of Automorphism. The Data Vault modeling structures are built The Data Vault Model is actually based on many different components of temporal mathematics and spatial mathematics. (I've listed a few of the research papers I used in the 1990's to help me construct the structural integrity of the Data Vault):

1. “Unifying Temporal Data Models via a Conceptual Model”, http://www.cs.arizona.edu/~rts/pubs/ISDec94.pdf
2. “Notions of Upward Compatibility of Temporal Query Language”, http://www.cs.arizona.edu/~rts/pubs/Wirtschafts.pdf
3. “Temporal Data Management”, http://oldwww.cs.aau.dk/research/DP/tdb/TimeCenter/TimeCenterPublications/TR-17.pdf
4. “Spatio-Temporal Data Types: An Approach to Modeling and Querying”, http://web.engr.oregonstate.edu/~erwig/papers/MovingObjects_GEOINF99.pdf
5. “Formal Semantics for Time in Databases”, http://portal.acm.org/citation.cfm?id=319986&coll=portal&dl=ACM&CFID=6511873&CFTOKEN=58729889

The Data Vault model is capable of adapting, changing on the fly, exhibiting the mathematical properties of automorphism, in that through architecture mining combined with data mining efforts we can "learn" what architecture flaws exist, where stronger relationships exist, and where the architecture can change itself or re-connect to itself to form a stronger data model.

How does this work?
The Data Vault LINK is made up of vectors. It houses a directional connection to each HUB that it is associated with. The vector of that connection can be assigned a strength and confidence co-efficient to determine it's usefulness within the data set contained within a link. Mining the data over time can produce a powerful combination of patterns of change, along with the discovery of patterns of association (possibly never known before), or as a result of a pre-known state.

The data mining tool can then be taught either "what to look for", or it can be set-off in discovery mode to associate information based on a Data Vault model already constructed (use the existing Data Vault model as a starting point for the learning process), and then it can determine if any "undiscovered" relationships exist. Furthermore the process of mining the data can then be used to assign strength and confidence coefficients to EACH of the vectors in each link, thus preparing for the architectural mining phase.

So how is the Data Vault automorphic?
The Data Vault is connected (within itself) to itself via the Links and the vectors within the links. Each vector can be considered a component within the mathematical matrix of the automorphic functions. Then, the mathematics of "groups" and vector analysis can be applied to dynamically alter the matrix for a potentially different outcome.

Thus, new links can be constructed on the fly, tested, and then removed (if no real value to the human on the other end of the computer is perceived). They can likewise be constructed, and then old linkages can be removed to produce an auto - morphing data structure, something akin to self-correcting. I will NOT go so far as to say it's actually learning, because it (the computer) will still not understand the CONTEXT to which it's applying the changes.

This type of system STILL requires guidance, training, and tweaking from the operators in order to achieve the desired outcomes and modifications to the model that make sense to the business, even if the business itself is commercial or government oriented.

However, this type of system can be applied (easily) directly to the Data Vault modeling constructs in order to achieve a self-changing data store, something that appears to "point-out" different facts, or discover different unknown relationships without understanding what it has. The understanding part is still up to the human.

Ok, so how does this benefit business?
Well, if we can spot relationship changes automatically (on a simplistic level), or mathematically figure out POSSIBLE infancies in our business, then we might be able to adapt our business based on the information being collected (or in some cases, adapt the source systems to collect information WE MISSED that might be vital to our operations). The data sets and the architecture of the data sets can tell us as much about our business as the processes and the business models we use.

You can find out more about the Data Vault model (for free) at: http://www.DanLinstedt.com

Hope this is interesting,
Daniel Linstedt