We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Data Warehousing - Pushing the Envelope

Originally published September 29, 2004

Data warehousing started in roughly 1984, at least in its nascent form. The idea that there should be a different way of aggregating data and a different way of processing that data was (and still is, in some circles) a novel idea. 

For data warehousing to have come so far in such a short period of time without the support of a vendor, is pretty remarkable. Today, there are many vendors that support data warehousing. That’s easy to do because the total expenditures for data warehousing and DSS have grown dramatically. But in the beginning, there was just a concept, some books, some articles and some classes and seminars.

ADVANCING TECHNOLOGY

In order for data warehousing to have succeeded, it is interesting to note that there were several technological breakthroughs that had to occur. What were some of the technological advances that were needed in order for data warehousing to progress to today’s level? They include:

  • Hardware to store the volumes of data.  Data warehouses require enormous amounts of disk storage. If we were still using the disk storage we had in 1985, we could neither afford data warehouses nor could we operate them. The capacity would be simply too much.  
  • Software to process the volumes of data. Not only are huge amounts of physical storage required, but DBMS software is needed to handle those volumes. Looking at the DBMS technology of 1985, data warehousing could not have thrived. DBMS vendors have done an exceptional job in building data warehousing on a technology that was originally wasn’t created for a data warehouse. 
  • Hardware architecture to support the volumes of data. Not only were hardware and storage advances required, but hardware architecture advances were needed. SMP architectures are fine for small to modest amounts of data, but an MPP architecture and its other derivations or the creation of SMP clusters are required to handle this volume of data.
  • ETL software. Can you imagine all of the data placed in today’s data warehouses being manually entered? It would be like building the Goldengate Bridge without a crane. If data warehousing was ever to succeed in a large way, it was necessary that automation of code, for the purpose of extracting, transforming and loading of the data be created. 
  • DSS analytical tools. If you build a data warehouse but can’t access or analyze the data, it doesn’t matter. With a wide-array of Business Intelligence vendors who have first-class analytical capabilities, today’s world of the data warehouse is blessed with powerful, sophisticated analytical tools. 

These are just some of the technologies and advances that make data warehousing what it is today. Additionally, there are other technologies that are maturing which will take data warehousing to the next level. Some of these are: 

  • Near-line storage.  Data has its own life cycle and as the volumes grow certain portions of the data become inactive. It then becomes prudent to place a portion of the data on near-line storage. Not only does near-line storage cut the costs for the data warehouse, but near-line storage increases dramatically the overall speed and system performance of the flow of data in the data warehouse. 
  • Archival storage. In the old days, when something was archived that meant it was thrown away. In today’s world when something is archived it can be retrieved. Unlike near-line storage, archival storage is not an extension of the online database.  
  • Data warehouse monitoring. Data warehouse monitors look at data usage down to a gnat’s eyelash and serve a fundamentally different purpose than the traditional online transaction monitors. If you use archival and/or near-line storage you need one of these monitors. 
  • Metadata. For a variety of reasons metadata has always been the Rodney Dangerfield of technology. It is eternally on the back burner. But with the volumes of data, the diversity of processing, and the diverse locations that processing occurs, there is a great need for metadata today, even though it is never in a position of priority. 
  • Unstructured data management.  From the beginning, data warehousing has just been concerned with structured data. But there is a need for including unstructured data as an essential component of data warehousing. 
  • Virtual ODS. A major development, virtual ODS technology provides the ultimate in flexibility. In many ways the virtual ODS fulfills the promise of database theoreticians made in the 1960’s and the 1970’s. But in the business context of today, the virtual ODS occupies a powerful position. 

So almost as if by magic, technology has heeded the call of data warehousing and propelled it forward. As a result, enterprises world-wide are now leveraging the power of data warehousing. 

  • Bill InmonBill Inmon

    Bill is universally recognized as the father of the data warehouse. He has more than 36 years of database technology management experience and data warehouse design expertise. He has published more than 40 books and 1,000 articles on data warehousing and data management, and his books have been translated into nine languages. He is known globally for his data warehouse development seminars and has been a keynote speaker for many major computing associations.

    Editor's Note: More articles, resources and events are available in Bill's BeyeNETWORK Expert Channel. Be sure to visit today!

Recent articles by Bill Inmon

 

Comments

Want to post a comment? Login or become a member today!

Be the first to comment!