Blog: Krish Krishnan

The Rise of the Crowd - Part 1

Tue, 07 Feb 2012 18:15:02 -0700

In the recently concluded Super Bowl 2012, we all know NY Giants won the championship, but in the preceding weeks there was an increasing sentiment expressed on Twitter about Eli Manning and at the end we all know the result.

If you have read James Surowiecki's book titled The Wisdom of Crowds, there is a famous example of the power of the crowd demonstrated by Sir Francis Galton. The story goes In 1906, he was visiting a livestock fair in England, where he stumbled upon an intriguing contest. An ox was put on display, and the villagers were invited to guess the animal's weight after it was slaughtered and dressed, paying 6 pence to participate. Nearly 800 people participated, but not one person hit the exact mark: 1,198 pounds. Galton collected the answers and applied the statistical mean of these guesses from independent people in the crowd: Astonishingly the mean of those 800 guesses was 1,197 pounds, accurate to fraction of a percent. This marks the first of the series of experiments conducted by scientists to prove the collective intelligence of the crowd.

What this proves to us is when you apply a set of smart people to solve a problem, any problem, chances of a solution are very more possible than a single person trying to do the same. Today the same type of contests are held by companies such as Kaggle, 99Designs, Innocentive, CrowdAnalytix and many others, where statisticians and analytic experts compete to solve such problems.

What is the use of these contests and these business models? well there are several benefits

The problem can be solved better by a crowd where it can be solved faster
The open innovation platform provides you access to more experts than any consulting expertise can provide
Costs can be better managed in an open contest where the solution has a fixed price and timeline

And the list goes on. We will see how challenges arise in this subject in tomorrow's blog

The topic is deep and wide, next week at TDWI Las Vegas, there is a night school session on this subject that I'm hosting, feel free to attend.

The Big Data Database Saga Continues

Thu, 19 Jan 2012 21:38:31 -0700

By now all of you have learned about the announcement from Amazon about DynamoDB, the latest database with NoSQL+Cassandra+Voldemort+Riak and a lot of other tools thrown together, completely hosted on the cloud, with the feature to scale on demand, a true elastic scalability similar to EC2. throw on top of this a MapReduce interface and you have a Big Data Database that can truly scale.

What sets DynamoDB in my simple tests over the past few hours is the simplicity that it brings to Big Data processing. While my tests are not complete yet, initial results are definitely encouraging. As I write this blog, I have also read Datastax's comparison of Cassandra and DynamoDB at - DataStax questions DynamoDB's performance. The comparison is long post full of technical comparisons around operations per second, but does not mention cost or services provision of DataStax. If you look at cost, Amazon says the services start at $1 per gigabyte per month. Data transfer is free for incoming data. It's also free for the first 10 terabytes per month and between AWS services (like Elastic MapReduce and S3). Once you surpass 10 terabytes, taking data out of the service is $0.12 per gigabyte through 40 terabytes and then lower rates up to 350 terabytes. Throughput capacity is $0.01 per hour for every 10 units of write capacity and $0.01 per hour for every 50 units of read capacity.

Based on where several internet-based, service companies have built models and found success, they will not have any hesitation in adopting to the DynamoDB platform. Especially with the ability to dial-up and dial-down scalability, you can really control costs, which even on a consistent basis will be much lesser compared to on-site provisioning for these companies. DynamoDB has beta clients like Elsevier, Formspring and SmugMug, which are definitely encouraging names.

As an organization, If one were to choose a cloud based services provider for Big Data, Amazon sounds a logical choice based on several fronts, but is your big data initiative internet deploy-able? and do you have staffing to execute the program even if you host the data on the cloud?. While you digest more content apart from this blog on DynamoDB, I will revert to running more experiments and share more information in the next few days on scalability tests and consistency of the database.

There are several NoSQL databases to compare DynamoDB against too for a fair comparison at the DB level.

Watch for further information on specifics.

EDW - It will be the Enterprise Data Warehouse

Wed, 11 Jan 2012 21:59:26 -0700

Recently a tweet caught several people's attention - "Eventually, Hadoop will swallow the EDW". Let us be very clear, the EDW will be needed now and in the future. The premise of an EDW is for processing and storing data for consumption across the Enterprise for Analytical and Reporting purposes. Hadoop is a platform for managing the processing of Big Data, it is not a relational data store and nor is it engineered to replace the EDW. Several people have a similar misconception, but Hadoop and EDW are mutually exclusive platforms and they will be integrated via strong Metadata relationships.

It is true that Hadoop is getting several upgrades and new distributors, but this does not mean you can move all your EDW data into that platform. Structured data is best processed on RDBMS platforms.

You can argue that one needs a hammer to drive a nail into the wall, but what type of hammer, what type of nail and what type of wall, all of these matter.

There are several articles in the internet including presentations from Hadoop community on why EDW. I urge you to do some research and understand the same. Plan on attending TDWI Las Vegas or Chicago this year to learn more on this, or plan to attend Enterprise Data World 2012 in Atlanta. We have several discussions and sessions on this subject.

Bottomline, EDW is here to stay and is nor getting retired soon.

Patterns

Wed, 04 Jan 2012 21:39:34 -0700

At a recent event where I did a keynote, an audience question was on why Big Data means processing with Patterns?. Let us take a step back and analyze this thought, Patterns have always been the way we have learned. Whether it is languages by symbology or music patterns, the human mind can imbibe those patterns and reproduce them. This concept extended to computing too, where we reduce different types of data ino binary symbols that are interpreted by the system.

The patterns are what we formed into thoughts and behaviors that manifested into Big Data, and it is the very same patterns that need to be disambiguated with context. If you draw full circle, patterns play an important role in any aspect of data processing.

Pattern processing is intricate and definitely complex, but there are robust techniques to accomplish this subject. With the advent of Parallel processing techniques for large scale data, Pattern based processing has become more scalable and flexible.

While the subject is not new, thinking about processing complex data from this perspective will be one approach to tackle the problem of Big Data processing

Unstructured Data - Complexity

Fri, 30 Dec 2011 09:16:58 -0700

Taming the three big things in Unstructured Data (Big Data) include Volume, Velocity and Complexity. While we can see infrastructure growing to handle the volume and velocity equations, the third and the most toughest task involves taming complexity.

Complexity comes in a variety of shapes and sizes within the unstructured world. The reason for this arises from the fact that all things textual, audio, video and more, are based on Human Reasoning and Thinking. The fundamental concept behind human reasoning relates every piece to a context, for example - you go to nice restaurant and order food, more than the food, you relate the restaurant to an occasion, people who you were with, date on which you went there. Assume that you will write about the food experience, your document will contain just more than pure food. If we were to process this as data, without the relevant context it is pure noise with hidden layers of complexity due to the different patterns of thoughts that have gone into the document.

If we were to now take a look at everything we do, without context we are lost. Hence the need for a robust set of contextualized rules are needed to process data in the unstructured world. Textual ETL is one such rules engine that can solve the complexity equation. You can also do the same in Java and MapReduce, though it is very laborious.

Social Media - Does it really influence?

Sat, 24 Dec 2011 10:49:35 -0700

Recently a study was done by a leading university, that concluded Social Media provides minimal influence on behavior etc (read it here - Havard Study). My issue with such a study is the perception and viewpoints do not account for all that is happening in reality, and the study is focused solely on Facebook. IF you need a prime example of Social Media and its influence, look at the powers that reshaped the political landscape across the world this year, Time Magazine naming the Protester as person of the year, and this is a collection of all persona's and personalities digital and physical.

Social Media has become a reckoning force, a vital tool for information exchange and to a large extent has fostered many startups hoping to cash on building platforms and services around the subject.

With enterprise Social Adoption becoming the biggest trend among large corporations, it is pitiable that we are still grasping at the straws on the influence factors and its value.

In my opinion, the question of value is really based on the underlying purpose of the social network. It relates to the goals and direction a community, based on its common subjects of interest and intent. If you want to derive the value quotient from this community, you need to study its behavior and its relationships amongst members of the community. You can use many algorithms for such purposes, though there is no set methods available as commercial or open source solution. You can definitely use technologies like Hadoop, Cassandra, Mahout, R, Textual ETL to create solutions that will help in driving analytics which can help you to create, define and measure the value of a social network.

Social Media Metrics have been a nascent area and are still emerging from a concept to a solution. It is immature to think because I do not know how to measure, I will rather assume the exercise is futile. Rather look at the whole movement behind Mahout and R, this area will emerge in 2012 as the most adopted solution platform.

Big Data - Worth it or not?

Thu, 15 Dec 2011 20:38:26 -0700

I'm getting a lot of questions from audiences about Big Data - what is it? Hadoop - what is this? and more importantly why do i need to understand this now? will it be needed for my organization?. Bottom line is that the user community is completely befuddled with the whole Big Data and Hadoop messages. Let us take a step back and clear the slate.

What is Big Data? in a true sense, data that has been used to make decisions with respect to your business - transactional, spreadsheets, emails, campaigns, sales-force, call center, competitive intelligence, analytics, legal contracts, manuals, web data, consumer forums, content management systems and more form the foundation for Big Data. Another dimension to view this is data that lies outside of transactional and EDW systems, that influence your business decisions and provide you insights into the consumer and product behavior is also called Big Data, as it has no defined structure or storage mechanism. Big Data is big in terms of size, volume and complexity independent of Time.

Now let us come to Hadoop, it is a software solution framework, distributed by Apache foundation. The architecture of Hadoop lends to distributed and parallel computing techniques, by which you can manage massive volumes of data and process it to harness the underlying value in a relatively manageable time. Hadoop is an ecosystem as it is a community developed project and has continued contributions from the community of developers.

Is Big Data worth pursuing for any organization? very much yes. But like any other solution you need a strong business case to implement this type of program. You need a very strong governance model to analyze the type of data you want and what business value will it bring to the organization. It is a maturity journey, and it is not a turnkey solution to switch on a technology from any vendor and declare victory.

Do I need to use Hadoop and learn everything it has as a framework? well if you have to yet start the Big Data journey, all RDBMS and DW Appliance vendors are working on a race against time to bring Hadoop integration via your favorite DB platform. But even after that integration is complete from the given vendor, you still need to understand how Hadoop works and what type of problems will it solve, you will need to understand TextualETL and how you can write business rules in English to process text data of any type and much more. But the silver lining in this cloud of complexity is, all these technologies are evolving and will mature by the time you are ready to adopt them. They are all GUI driven and self managing, they all address problems that your current RDBMS or EDW platforms cannot resolve.

Two words of advise, Learn everything you can about these topics, sort out Blather and Real Information on these topics.

Remember not every business is Facebook or Twitter or FourSquare, but every business will evolve in the future to adopt similar models to emerge Customer Centric. As you start this journey, remember it is a Continuum and has multiple stages of maturity.

The New Age BI

Tue, 06 Dec 2011 21:02:57 -0700

AS we wind down 2011, the year has been the breaking ground for "new age BI". The change in BI from traditional to new is in the following respects

Agile - the demand to provide reporting and analytics on demand on the most recent data.
Mobile - the report and analytics must be supported on mobile platforms.
Multi-Sourced - the reports and dashboards should be able to integrate data across sources. Needs a strong metadata footprint to support this.
Self-Service - the new BI tool should support self service reporting and analytics
Light Weight - the BI tool should be light weight and have a nimble footprint
Apps like capabilities - must be capable to run and deploy as Apps (as in iPad and Android Apps)
Built in support for Office or Open Office - the BI tool must have support for
Office or Open Office apps
Unsrtuctured Data - the BI tool must support unstructured data and its requirements
In-Memory Capabilities

The list can go on and on, but these are a driving shifts that will move BI to the next generation and in 2012 will define the course for the leading platforms in this realm QlikView, Spotfire, Tableau, Microstrategy Mobile. Organizations have already started using the new platforms as augments to the existing platforms such as Cognos, Business Obejcts, OBIEE and Microstrategy. The most easily adopted tool by business users is Tableau and Spotfire in the current trend and enterprise users have started taking a hard look at Qlikview.

2012 will be a new year for BI and will probably kick off the new decade for BI as well with the new tools and trends. Only time will tell. But the future is here and being explored as we read this.

What is the value of my Data?

Fri, 02 Dec 2011 06:00:00 -0700

An intriguing bemoan from most users that I have heard is "what is the value of my data?". Well if you are looking for $$s and cents, then your data has value from the time it is created till the next few hours where it gets analyzed, after that it is history and considered as dead weight in strict terms. However being born collectors, human tendency is to keep all this weight around and get a very obese and unhealthy data collection, that will eventually just die due to its sheer volume(read weight).

The value of your data is measured in different ways

Origination - This is a point of creation of the data, can be a transaction, an email, an application for insurance or a claim. Data is deemed to be often dirty at this juncture and data quality rules are applied for correction. This is the point in time where data has the highest value
Transformation - This is a point of collecting and transforming the data to be ingested into analytical and reporting platforms. At this point again due to the number of rules that are applied, data here will have a very high value
Analysis and Reporting - This is the last point in the life-cycle where the data value is held high. The data here points to trends and behaviors as simple metrics, but yet will serve a very useful purpose of being a treasured indicator

There are a number of Data Quality indicators that will be able to measure the effectiveness of quality across the enterprise both in origination and transformation phase. These indicators will be a very useful point to prove that data of good quality has a high value as it helps speed up decision support platforms. This is one way to assess and prove the value of your data.

The second way to assess and prove the value of data, is to measure its effectiveness when used as metrics and KPI's in reports and Analytics. The quality and timeliness of data here will be measurable with future results which can be compared with current results, and the differential lift can be attributed to the value of data being available, with the right quality and at the right time.

While none of these are new techniques, with the same question arising many times, it is prudent to nudge the good old ways and ensure that the simplicity of using these can be the best innovations that you have accomplished in assessing the value of your data.

Watch Out - Data Storm Headed This Way !!!

Wed, 30 Nov 2011 09:24:42 -0700

Pretty soon we can start having Weathermen saying alerts such as "Data Storm" or "Data Blizzard" is headed towards cities and associated details, and I mean this with all seriousness. The bizzare volume of data growing in the public domain is just too much to comprehend for a human mind. The frenzy for generating data comes with continuous innovation in mobility, wireless communications, stronger infrastructure and an ever ready audience to listen. If you want to know what is happening in a particular celebrity's personal life at any given moment, you can find a million social media sites, forums and more; if you want to know where the next internet led revolution will start, you can keep tuned into multiple facebook pages, multiple social media sites. The point is as facilities increase in scalability and technology is more affordable, you will have more data producers then data consumers, and more relevant will be the question of "what is noise and what is real information"?.

Data generation is not a bad idea, but who uses that data and how useful can that data be is a question of relevance. If you are on Facebook and post every hour to your page, chances are someone in your primary or extended network will be watching it or posting on your wall or something. Those critical transactions are needed for FB to run ads and look at influencer behaviors etc, they really do not care about the personal side of the data at all.

Now lets take a look at a consumer's own behavior,for example ( to make it easy) how much of value do you see from data in your FB pages, and how much do you care about the data, say after one day, week, month or year ? you really do not think about it, as you are using the platform for free (actually paying for it in some very indirect way). This is why the need to look for Data Storms in the future.

The information overload from the public domain data that is produced for content sharing and sentiment sharing is causing a tizzy. There is value for points in time in this data, but where is it stored? who needs it? and does it make sense to keep it? what are the security threats for this data? Thus comes the big question of "Data Governance for Public Data". Sooner or later, we need to address this question and guess what I recommend that it be a decision that is driven and defined by the same consumers who generate the data everyday, as it is about their information privacy, security and more.

But till then, let's watch for weathermen to announce Data Storms or Data Weather Readings.

The Complexities of Processing Big Data

Mon, 21 Nov 2011 07:21:57 -0700

Over the weekend, I participated in the BigDataCamp in Chicago and met with over 100 people, spending a Saturday to understand what is the Big Data and Hadoop/NoSQL stuff about. What is very interesting to learn and observe from this experience and from the other engagements of last six months is, most people have to yet grasp the complexities of processing Big Data. While technologies such as Hadoop, Mahout, Cassandra, HBase, Hive and more will help you start, there is a lot beneath that surface.

The world of Big Data will need one of think of developing Algorithmic approaches to problem solving. This is no more a drag and drop world of processing data through fancy GUI's, that evolution is being attempted by vendors. There are reasons of why you need to slow down in ingesting this type of data, and bite off complexities in incremental chunks

Big Data is in several structured, semi-structured and unstructured formats. Most of the data in this world is buried in context and content does not lend itself easily to process without context.
Big Data cannot be tokenized very easily, this is true when you start thinking about end results from the process and the associated KPI's that you want to derive. The tokenization will need two parts - a robust metadata and a strong taxonomy.
Big Data will need several data quality rules to be implemented, which means you need extensive data cleansing and processing. Beyond English, there are several languages that you will need to deal with.
There are no set paths to process images, videos and more complex data. These are still evolving and need more time to mature
The co-relation and derivation of meaning from compressed terms is not simplistic for any non-business SME.
There are different types of content that need multiple rules to process them, derive the actual underlying meaning and then prepare them for integration
Taxonomy driven navigation is a good start, but often not enough to create the appropriate context.

While the platform for processing is ready and available, there is no simple process to walk the complexities. A couple of good solutions that are out there are Textual ETL (Bill Inmon) and the Mahout projects (Machine Laarning from Apache community). The Textual ETL engine is a business facing rules creating tool that helps process Big Data, Mahout is more complex, but had ready-made algorithms for Recommendation engines and more. Feel free to look at both these solutions, as they solve different problems and are not competing solutions but complementary solutions.

As I had mentioned earlier, remember the Big Data world is fascinating and provides lot of valuable insights, but the hidden complexities of processing and integrating this data, to make it meaningful will make it seem a daunting task or even a over-engineering solution build.

Economics of Social Media Data

Sat, 12 Nov 2011 19:50:32 -0700

I have worked on several "social BI" projects over the past two years, some have been extremely successful, others have had their fair share of adoption issues. The bottom-line that hits home is the fact - is this program worth all the effort?

Well, if one were to ask this question throughout the program, the answer will be "yes", and if you wonder why?, there are several reasons

Social Media has rich content of consumer sentiments, brand image and sentiments, brand reach, consumer awareness, competitive information, research data and more
Social Media has information on patterns of behaviors of products, markets and consumers over time across the world
Social Media has several independent venues providing critical research on anything under the sun
Social Media also has the most important digital presence - Your Organization and Your Customers

If one were to harness all the data of these types and more, and establish the right meaningful content disambiguation rules applied to the same, it would provide rich information that can be used in conjunction with your internal systems and provide you a collage of behaviors, events and triggers that lead to your product and its adoption, your consumers and their sentiments, and much more.

Now let us talk economics, lets say your average marketing spend is $500k per year for the enterprise, and your line of business is home delivery of groceries. Your business has slowed down and you hire a market research company to assess why and what?. Let's assume they come back and recommend you increase market spend and also start looking at new locations for additional market, would you spend the $$'s to increase marketing?.

Not yet, because through additional research of your own if you find that there is competitive threats, consumer sentiments about your decline of quality of service, you now want to learn more. So you decide to spend say $250k to integrate social media data and you want twitter, forums, facebook pages as your target sources. In harvesting the data feeds through various listening posts, you discover that

You have lost 20% market to competition due to service delivery issues
You have lost 8% of your clients due to economic conditions
You can gain additional clients in remote areas if you can expand services for certain brands of foods
Your preference of foods brands have created a group of loyal followers, who can be your Word Of Mouth Marketing if you can cater them well, and their clout will bring you a potential 30% net new clients
Your products need revamping
Your Call centers need more data to help and support beyond just orders
Your ability to cross sell and up sell was limited and customers left for more lucrative offers

Upon getting better insights, you now change your product strategy, your CRM improves drastically, your quality meters go up and most importantly, you start developing a "crowd" of customers who become your ambassadors and advisers.

Now let's say you spend the new dollars of $200k on marketing, your sales jumps by 5% of new clients, 10% of customers were given additional options in cross sell and up sell, 40% of your customers give you a "AAA" rating, a new lift of 18% from prior.

In this situation, the economics of social media data is seen very clearly. There are more such instances that can be discussed and business cases articulated.

My two cents, there is lot of rich and varied content available, you need to decide what is best for your organization, if you want to compete and remain in business in the next decade, you will adopt to Social Media strategies and the economics of social media data will always provide measurable gains. You need some software tools like Spotfire from Tibco and Tableau to help create powerful dashboards and give you abilities to drill down to better insights.

There are a number of technologies and data integration points that need to be designed and implemented, that will be a whitepaper that will be available in December 2011 for your reading in my channel.

The new age of Data

Mon, 07 Nov 2011 20:01:27 -0700

Like the history of Earth, data has history too. We had the ICE age of data which was the punch card and large magnetic tape days. Data was primitive and binary in nature. Then we matured and came to stone age of data where we had mainframe computers and started computing with languages such as COBOL. Along came PASCAL and FORTRAN. Data has started becoming mature and available for consumption.

Next came the iron age where we discovered UNIX and C (Thanks to Ritchie and Thompson), we were suddenly capable of building strong processing platforms and found that we could process data on the mainframe and Unix as well. Data was fast emerging from the shadows as commodity in the enterprise We started seeing the convergence of Relational theory and saw the glimpses of the first relational databases.

Next came the bronze age with the advent of Macintosh, OS/2 and Windows. We discovered client-server computing, wow what a novel ides, along with client/server came networking, security, emergence of computing platforms such as Visual Basic and Power Builder. We could now build smart applications, tie them to powerful back-end DBMS systems such as Oracle, Sybase, Informix, DB@ for UNIX and more. Data was fast emerging as the backbone of enterprise applications.

As we moved out of Bronze age of data, we were already swimming in ERP, Supply Chain, Logistics, Transportation, Warehousing and CRM. This was already causing consumers of data to go dizzy.

And now we are stepping into the new age of Data - BIG Data. Data has grown in size and volume as the name indicates.

This new age of data is ushering new age of possibilities, we are now able to visualize everything from behavior of people on websites to what drives communities and interests by following the clickstream. We are able to see crowdsourcing in action on the internet. We are able to get insights like never before on patterns and what drives them. As we start swimming in this fast moving current of data, it is also becoming clear that we will innovate faster than ever before in terms of solutions and applications with this new found capabilities. This new age of data will create a market and opportunity like never before. In reality we can say that it is the coming of age of the internet.

As I'm writing this blog the silicon valley VC community is pumping money into ventures that even spell Big Data or Hadoop remotely. This kind of action was last seen in the early 90's. We are probably well into the next boom period and this will be a long tail boom, meaning volume driven.

Big Data - It's not Tinkering, it's Transformation

Fri, 04 Nov 2011 15:58:36 -0700

The buzz and hype is going around you everywhere, whether it is a leading technology publication or a vendor or any website related to IT or any conference, the keyword is "BIGDATA". Look at this wordcloud

Any search on the topic brings technologies, companies and more.

Lets get real, when you start talking a BigData project, it is a transformation for the enterprise. You are not talking about just another data project in the enterprise, it is not another Data Warehouse or another Application. It is a combination of all things that you have in the organization about data, analyzing and presenting the results in dynamic dashboards, mobile technologies and more.

A BigData project is not about tinkering with the hottest technologies, rather it is an innovation and disruption (internal and external) that you want to bring to the enterprise. The success criteria is not how many petabytes you have, but what influence did the insights from the data provide to you as an organization and the benefits it can bring to your business and its customers. Think different, because you are dealing with something that has never been done in the enterprise prior to this program.

Another key to remember is the success of any BigData transformation will start with champions among the people in your organization. We are seeing the emergence of a new role for these champions - Data Scientists.

While technology is important and a lot of it is new and emerging, the people and process side of this transformation is very key to make it a valuable asset for the enterprise. How do you monetize on the data that you have accumulated in the enterprise? it gets better, how do you monetize on social media data that provides rich customer sentiment and also enriched competitive intelligence? all these are the major goals for our own BigData program.

Bottomline to ensure is you have the executive sponsorship and for that your executive will need to understand that it is a transformation and not a tinkering exercise.

Watch this blog and twitter for announcements on technologies, frameworks and implementation methodologies, all for your BigData success.

The BIG Data Confusion

Mon, 24 Oct 2011 22:07:06 -0700

There is a widespread confusion in the market with this buzzword or catch phrase - BIG Data. If one sits through the presentations from 10 vendors, 20 different definitions are likely to come forward. Each definition, is skewed to support that vendor's products or services.

This is where the issue is. I have had people tell me they work on BIG Data and then talk about the volume of data and not the type of data. There are people that talk about BIG Data and then focus on CLOB and BLOB text. There are people who are implementing Sentiment Analytics tools and have said they are doing BIG Data.

To simplify the confusion, lets look at the world this way -

1. Companies need to make business decisions and compete on products and services, to accomplish this they build operational systems, transaction processing systems, CRM, ERP and SCM systems. All these systems create and generate data from different touchpoints. All this data is then collected in an ODS and EDW/Datamarts to generate reports and be used in analytics. Once the reports and analytics are done, the data is consumed by business managers and executives who take action. All the action is happening on emails, documents and is not related back to the EDW or ODS. The data generated by all the systems including email, documents and operational systems collectively form one form of BIG Data - internal Big Data.

2. There is external BIG Data - Clickstream logs, machine generated logs, 3rd Party, Sentiments, Forums, Blogs etc, which need to be analyzed for Competitive Intelligence, Voice of Customer, Behavior Trends, Speech to Text (Call Center) and Image processing.

All of these types of data when used by any company will be the true definition of BIG Data. Hadoop is one platform to solve this problem, but there are more emerging and semi-established platforms too. All major vendors in the DW space have already pledged support to Hadoop and talk about BIG Data specific solutions.

This blog provide you my perspective on BIG Data definition. You may or may not agree to it, but this is the real deal based on experiences from a few BIG Data projects that I worked on in the past few months.