We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.


Blog: Krish Krishnan Subscribe to this blog's RSS feed!

Krish Krishnan

"If we knew what it was we were doing, it would not be called research, would it?" - Albert Einstein.

Hello, and welcome to my blog.

I would like to use this blog to have constructive communication and exchanges of ideas in the business intelligence community on topics from data warehousing to SOA to governance, and all the topics in the umbrella of these subjects.

To maximize this blog's value, it must be an interactive venue. This means your input is vital to the blog's success. All that I ask from this audience is to treat everybody in this blog community and the blog itself with respect.

So let's start blogging and share our ideas, opinions, perspectives and keep the creative juices flowing!

About the author >

Krish Krishnan is a worldwide-recognized expert in the strategy, architecture, and implementation of high-performance data warehousing solutions and big data. He is a visionary data warehouse thought leader and is ranked as one of the top data warehouse consultants in the world. As an independent analyst, Krish regularly speaks at leading industry conferences and user groups. He has written prolifically in trade publications and eBooks, contributing over 150 articles, viewpoints, and case studies on big data, business intelligence, data warehousing, data warehouse appliances, and high-performance architectures. He co-authored Building the Unstructured Data Warehouse with Bill Inmon in 2011, and Morgan Kaufmann will publish his first independent writing project, Data Warehousing in the Age of Big Data, in August 2013.

With over 21 years of professional experience, Krish has solved complex solution architecture problems for global Fortune 1000 clients, and has designed and tuned some of the world’s largest data warehouses and business intelligence platforms. He is currently promoting the next generation of data warehousing, focusing on big data, semantic technologies, crowdsourcing, analytics, and platform engineering.

Krish is the president of Sixth Sense Advisors Inc., a Chicago-based company providing independent analyst, management consulting, strategy and innovation advisory and technology consulting services in big data, data warehousing, and business intelligence. He serves as a technology advisor to several companies, and is actively sought after by investors to assess startup companies in data management and associated emerging technology areas. He publishes with the BeyeNETWORK.com where he leads the Data Warehouse Appliances and Architecture Expert Channel.

Editor's Note: More articles and resources are available in Krish's BeyeNETWORK Expert Channel. Be sure to visit today!

August 2012 Archives

One of the key requirements for the next generation Data Warehouse is the ability to process streaming data and provide analytical insights while the data is in transit. The technology to process such a requirement exists outside the DW today, and there are platforms that have been enabled to provide these types of features within the DW infrastructure by IBM, Teradata and other vendors. The real question is does the business have the data in their hands at the right time? and an even more pressing question is why does the DW need to process this type of requirement?

Let us look at the first question, today the business does not get access to the data at the right time, for example credit card fraud, high volume trade fraud or any risk management platform can process a fraud alert based on complex event processing algorithms, but the results are provided to the business user in most cases after the fact when the data is at rest in the DW. This now adds complexity as someone has to trace through the data end to end to sequence the events and discover patterns and with add the added delays due to human intervention in the process, the decisions are often more impacting on the financials. In the present infrastructure, applying a near real-time processing platform is expensive.

Let us look at the second question - why the DW? the answer to this question lies in the fact that the DW is the most stable data repository and the most integrated analytical data source in any organization. The deeper question is should this type of processing be done in the DW or outside the DW with just the results being integrated to the DW. Doing it the latter way is ideal as the volume of data we add is considerably lower compared to all the data being loaded and processed in the DW

This is where the next generation of DW comes into play. In the future, the DW will be the analytical hub with an ecosystem of technologies surrounding it. The better designed platforms for such streaming data analytics include Hadoop, NoSQL and Armanta Inc's platform to name a few. These platforms can process and digest high volumes of data and provide alerts that can be caught while the transaction is in process. An example of such an implementation is where a large credit card vendor can provide an additional client verification service and reject the transaction if found or suspected to be fraudulent.

As businesses transform to a service centric model, these types of data management needs will become the reality of the day. While there are several niche solutions and initial adopters to these types of platforms, the immediate future looks promising to lap-up all the new capabilities especially in the Big Data platform, where cost is not the barrier to enter.

Streaming Data Processing and Analytics will emerge as a top requirement, from a large market demand and adoption perspective in the world of Data Warehousing.

Posted August 28, 2012 6:41 PM
Permalink | 1 Comment |