Blog: Richard Hackathorn

« Private Data Sold by India Call Center | Main | Importance of Enterprise Compliance »

What to do about Very Large Data Streams (VLDS)

In Colin White's article on Enterprise Data Mountain, Part I, he opens Pandora’s Box of various data store components for the new architecture of enterprise data integration, whose objective is to create an integrated and consistent view of your enterprise. The architecture is service-oriented using an Enterprise Service Bus to link the components.

Forget the old paradigm of OLTP-OLAP happily coexisting! It isn’t your father’s data warehouse! It’s a new ballgame, folks!

But... Colin forgot an important component... More and more data comes to the enterprise as a stream. You say, "So what? Just batch the stream!"

This is a BIG stream, as in megabytes per second coming from a virtual data store of many petabytes. This is a CONSTANTLY CHANGING stream in content and structure. Forget storing the data. Examine it on the fly and decide instantly what aspects are important to manage within the enterprise data mountain. This gives new meaning to the phrase "unstructured data". See the work of Jeff Jonas of IBM on perpetual analytics.

Let’s label this new component as a Very Large Data Stream (VLDS), as a bow to the critical contributions of the Very Large Data Base community over the past twenty years.

Is VLDS important? I would predict that such streams are and will be critical sources to the enterprise data mountain as it evolves of the coming decade. New media sources (established news channels, blogging, YouTube), near real-time satellite images, detailed weather data, point-of-sale data from all your stores plus samples of the rest of your industry, RFID data tracking your full supply chain globally, and on and on... That is Enterprise 2.0 of the future.

What do you think? Crazy ideas or right on?

Share: