(This is part of our ongoing Series of Unfortunate Data Warehousing and Business Intelligence Events. Click for the complete series, so far.)

A fundamental flaw of many business intelligence solutions is recreating what the company is already using for reporting and analysis. This takes one of two paths:

1)    The data warehouse is built using essentially the source systems' data model.
2)    The other end of the spectrum from 3NF is recreating your current reporting solutions, often data shadow systems or spreadmarts, that basically flatten out the data.

Click to read the complete post at the Data Doghouse.

Over the two decades that companies have been designing and implementing business intelligence and data warehousing solutions they have made the same mistakes over and over again.

Who makes these unfortunate mistakes? Newcomers, mostly. But don't be surprised to see experienced practitioners, who made these mistakes as newcomers, keep making the same mistakes.

Click to read the rest of this blog post on the Data Doghouse.

It's a vicious battle. No, I am not talking about the partisan wars between the Democrats and Republicans, but this battle also impedes progress. Nor am I talking about the decades- or centuries-old rivalries between religious or ethnic groups across the globe, but this battle has been going on within enterprises since their first use of computers.

The battle is between the curious and the cautious regarding the use of technology. The curious would call themselves innovators while the cautious call them reckless. The cautious consider themselves prudent while the curious call them Luddites.

This battle isn't just being fought on the fringes of the technology adoption curve. It's raging smack in the middle of mainstream technology - including technology that's already become a best practice within Fortune 1000 companies.

>>>continue

I'm not only concerned about hand-coding versus ETL tools;  I'm also concerned that potential buyers of ETL tools and the market in general are only looking at a small number of players in the ETL marketplace.

For many years industry analyst research groups have identified the top two product vendors: Informatica and IBM (from its acquisition of Ascential Software). So, naturally, these two appear on any evaluation shortlist. The rest of the evaluation shortlist usually includes the bundled products (mentioned in my recent posts) that come with the databases, BI tools or applications that the evaluating company already owns. Beyond these usual suspects, other ETL or data integration products are pretty obscure and almost invisible, at least from a general market perspective.

>>>Continue this post on The Data Doghouse

This is a continuation of an earlier post that discussed the problems of hand-coding using ETL tools.

What Went Wrong?

There are two aspects of effectively leveraging an ETL tool. First is learning the tool's mechanics. e.g. taking the tool vendors' training either in a class or through their on-line tutorials. Most IT people have no problem learning a tool's syntax.  Since they most likely already know SQL, they learn the tool very quickly.

But the second aspect actually involves understanding ETL processes. This includes knowing the data-integration processes needed to gather, conform, cleanse and transform; understanding not only what is dimensional modeling but why and how do you deploy it; being able to implement slowly changing dimensions (SCD) and change data capture (CDC); understanding the data demands of business intelligence; and being able to implement error handling and conditional processing.

>>>continue reading this post on The Data Doghouse

We are creatures of habit. It's not easy to stop doing something the way we've always done it. Especially when we think we are right (but actually we're not). Let's explain.

I have discussed (some might say preached) in many posts, articles, webinars, podcasts, classes and client discussions that for any recurring data integration tasks IT should use an Extract, Transform and Load (ETL)  tool.

This certainly has been the best practice for enterprise data warehousing projects in the Fortune 1000. This is where I got my early experience in data integration and got to use the ETL tools that annually rank in Gartner's Upper Magic Quadrant and Forrester's Top Wave. These ETL tools enabled IT groups and SI (system integrator) project teams to tackle data integration challenges too complex and extensive for hand-coding.

>>>continue reading this post on The Data Doghouse 

In our post Enterprise Software: Is there any one left to buy? we listed the best candidates to be acquired within the lucrative Business Intelligence (BI) and Data Warehousing (DW) industry:

• SAS (Private)
• Informatica (INFA)
• MicroStrategy (MSTR)
• Netezza (NZ)
• Teradata (TDC)

We have ruled out privately-held SAS because they are very reluctant to be acquired. (If they were willing then IBM would likely have bought them versus SPSS...no offense to SPSS but SAS is the 800-lb gorilla in the business analytics segment.)

The Usual Suspects

The high tech Titans - IBM (IBM), Hewlett-Packard (HPQ), Oracle (ORCL), Microsoft (MSFT), SAP (SAP) - are already very active in the DW/BI industry selling software, services and in some cases hardware. Are any of the acquisition targets on our list attractive to the Titans and would impact their market presence?

>>>Continue reading Enterprise Software: Who Should Buy Whom?

The high tech Titans - IBM (IBM), Hewlett-Packard (HPQ), Oracle (ORCL), Microsoft (MSFT), SAP (SAP) and EMC (EMC) - have been buying software firms at an amazing pace over the last half decade. According to BusinessWeek the tally is: Oracle $30 billion for 56 companies, Microsoft 79 companies, IBM 60 companies, EMC 40, and HPQ 34. Many of these acquisitions have been "tuck-ins" where the Titans have added the purchased company's product capabilities into their product portfolio. This is a time honored practice the Titans and their smaller brethren, have used to expand beyond organic growth and tap the innovative ideas of entrepreneurs.

Although tuck-ins have been a mainstay of software mergers and acquisitions (M&A) for decades, Oracle in the last decade has raised the bar by being successful in acquiring firms of a much larger size and, very much against software etiquette, even completing hostile takeovers. PeopleSoft and Seibel are the two case studies to validate this new approach.

>>>Continue reading this post on The Data Doghouse

A recent flurry of news articles in the database industry has reinforced my perceptions that the software titans have lost touch with current market needs. 

The PR machines from IBM and Oracle have been each claiming to have the fastest database and hardware platform on the market today. Database vendors battling each other on who has the fastest or the most features is nothing new and has been going on for a couple of decades. But what has changed is "who cares?"  Of course it does matter if you are a techie and you are evaluating which platform has those characteristics. It also matters to industry analyst and consultants that are evaluating and recommending the "best" database platform one can buy.  But maybe the key criterion for future database market share growth is not the best but what is the "best buy." 

>>>Read the rest of Database Wars: Fighting Yesterday's Battle?