We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Blog: Barry Devlin Subscribe to this blog's RSS feed!

Barry Devlin

As one of the founders of data warehousing back in the mid-1980s, a question I increasingly ask myself over 25 years later is: Are our prior architectural and design decisions still relevant in the light of today's business needs and technological advances? I'll pose this and related questions in this blog as I see industry announcements and changes in way businesses make decisions. I'd love to hear your answers and, indeed, questions in the same vein.

About the author >

Dr. Barry Devlin is among the foremost authorities in the world on business insight and data warehousing. He was responsible for the definition of IBM's data warehouse architecture in the mid '80s and authored the first paper on the topic in the IBM Systems Journal in 1988. He is a widely respected consultant and lecturer on this and related topics, and author of the comprehensive book Data Warehouse: From Architecture to Implementation.

Barry's interest today covers the wider field of a fully integrated business, covering informational, operational and collaborative environments and, in particular, how to present the end user with an holistic experience of the business through IT. These aims, and a growing conviction that the original data warehouse architecture struggles to meet modern business needs for near real-time business intelligence (BI) and support for big data, drove Barry’s latest book, Business unIntelligence: Insight and Innovation Beyond Analytics, now available in print and eBook editions.

Barry has worked in the IT industry for more than 30 years, mainly as a Distinguished Engineer for IBM in Dublin, Ireland. He is now founder and principal of 9sight Consulting, specializing in the human, organizational and IT implications and design of deep business insight solutions.

Editor's Note: Find more articles and resources in Barry's BeyeNETWORK Expert Channel and blog. Be sure to visit today!

Speeding up database performance for analytic work has been all the rage recently.  Most of the new players in the field tout a combination of hardware and software advances to achieve 10-100 times and more improvement in query speeds.  Netezza's approach has been more hardware-oriented than most--their major innovation being the FPGA (field-programmable gate array) that sits between the disk and processor in each Snippet Blade (basically, MPP node).  The FPGA is coded with a number of Fast Engines, two of which, in particular, drive performance: the Compress engine, which compresses and decompresses data to and from the disk, and the Project and Restrict engine responsible for removing unneeded data from the stream coming off the disk.  Netezza say that data volumes through the rest of the system can be reduced by as much as 95% in this manner.

So, the FPGA is the magic ingredient.  Combine that with a re-architecting of Netezza in TwinFin, released last August, that more effectively layered the disk access and moved to Intel-based CPUs on IBM BladeCenter technology, and you can see why Daniel Abadi came to the very prescient conclusion a month ago that IBM would be a likely suitor to acquire Netezza.

It seems likely that the short-term intent of the acquisition is to boost IBM's presence in the appliance market, competing especially with Oracle Exadata, not to mention EMC Greenplum and Terdata.  Of more interest is the medium- and longer-term directions for the combined product line and for data warehousing in general.  Curt Monash has already given his well-judged thoughts on the product implications, to which I'd like to add some.

My thoughts relate to the broad parallel between FPGA programming and microcode.  You could argue that Netezza's FPGA is basically a microcoded accelerator for analytic access to data on commodity hard drives.  As a long-time proponent of microcoded dedicated components and accelerators in its systems, dating all the way back to the System/360, IBM's way of thinking and Netezza's approach align nicely.  The question, of course, is how transparently it could be done underneath DB2, and further, the willingness of DB2 for Linux, UNIX and Windows to embrace the use of accelerators as DB2 for z/OS has.  The possible application of this approach under the Informix database shouldn't be forgotten either.

The interesting thing here is that the Netezza Fast Engine approach is inherently extensible.  The MPP node passes information to the FPGA as to the characteristics of the query, allowing it to perform appropriate preprocessing on the data streaming to or from the disk.  In theory, at least, there is no reason why such preprocessing couldn't be applied in situations other than analytic.  Using contextual metadata to qualify OLTP records?  Preprocessing content to mine implicit metadata?  Encryption / decryption?  It all lines up well with my contention that we are seeing the beginning of a convergence between the currently separate worlds of operational, informational and collaborative information.

But, what does this acquisition suggest for data warehousing in general?  Well, despite my long history with and high regard for IBM, I do fear that this acquisition is part of a trend that is reducing innovation in the industry.  The explosion of start-ups in BI over the past few years has resulted in a wonderful blossoming of new ideas, in stark contrast to the previous ten years, when traditional relational databases were the answer; now, what's the question.  Big companies find it very difficult to nurture innovation and their acquisitions often end up killing the spark that made the start-up worth acquiring in the first place.  IBM is by no means the worst in this regard, but I do hope that the inventions and innovations the characterized Netezza continue to live and thrive in Big Blue...  for the good of the data warehousing industry.

Posted September 22, 2010 3:02 PM
Permalink | No Comments |

Leave a comment