We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Big Data Projects Why are they such a big challenge for enterprises?

Originally published October 15, 2013

Many enterprises have had some experience with the volume aspect of big data (see my blog for big data intro), and many have a solution in place to handle it. But the new Hadoop-based platforms, which are further driving down the price point from a cost/terabyte or more likely cost/petabyte perspective, have introduced some significant opportunities as well as challenges. No doubt, enterprises have built a good level of expertise around managing structured data repositories, such as Oracle, DB2, large instances of Teradata and so on. But the Hadoop environment is a very different story. This is a completely new stack, and most enterprises have little to no experience with this platform. This puts enterprises in a fairly difficult situation.

The Challenges

Given the market conditions and the financial situation of most enterprises, only small budgets are being dedicated to prove the value of big data initiatives before committing to large scale, enterprise-grade projects to fully leverage the value. Most enterprises I’ve had the opportunity to talk with over the last couple of years are in some phase of initiating or carrying out a pilot project or proof-of value initiative to build the case for bigger budgets. But this journey has been a challenging one. I see it as the classic situation of the “teething” problems experienced with any new major technology adoption, and big data clearly fits into this category. All the clients I’ve had conversations with over the last couple of years appear to be in fairly similar situations.

Here are a few approaches I’ve seen them take and the challenges being encountered.
Hire: Some enterprises decide to hire new talent to get their initial pilot projects going – but big data skills are not a very readily available skill (yet) in the marketplace. As a result, this approach takes quite a bit of time and effort. It is a struggle to identify and screen for the right person with the technical expertise they need. This is an ideal opportunity for partners to step in, but even there, the expertise is not always readily available because it is in the process of being built up based on the market demands. So this causes a bit of a chicken-and-egg situation, adding further delays to getting big data initiatives started.

Some enterprises decide to train their internal staff. As one can imagine, training people to implement pilot projects does not always create/build the level of expertise required to create a high impact value proposition. In addition, a lot of time is expended in trial-and-error approaches, significantly extending the timelines.

: A few clients have adopted the outsourcing approach where they choose a partner to outsource their “first” pilot initiative. The challenge here is identifying a partner who has either done this before – a fairly scarce option – or, more often, a partner who is willing to bring the right expertise to the table and learn on the job. This partner is chosen to deliver the solution and to do it at a compelling price point, making it attractive to enterprises. Enterprises have a fairly mixed experience with this type of approach and so is not the most sought-after approach. However, in my experience, this has been the best and most effective way for clients to get a successful pilot completed. Yes, identifying the right outsourcing partner is critical step. There are several consultancies that are willing to take on such initiatives and share the risk. In fact, the few successes I have had (I can count them on my fingers) have been in this mode [apologies for the plug].

Identify the Business Problem

The second big challenge, or one can argue it’s the first big one, is to define the right business problem to target for the pilot. This again puts enterprises in a difficult situation because they do not have prior experience on what could be a good initiative. There is a lot of noise on the Internet on what type of initiatives people have taken up, but it doesn’t easily translate into ones that would be appropriate for each enterprise’s unique situation. Here again the options enterprises have tried are the same: hire, train or outsource. Again, my experience has been that the outsource option has been the one where I’ve seen the most success for enterprises. The right partners can help define the problem statement for these pilot projects. In fact, in addition to identifying the right partner, it is equally important that the right problem statement and success criteria be established before commencing such an initiative and that payments be tied to such criteria. The outsourcing option also provides a great learning opportunity for enterprises to ramp up and understand the possibilities.

Before I conclude, let me state for the record that a small subset of enterprises has made very significant progress and advancements with their big data projects, but the vast majority are in the early phase of figuring out how and what they are going to do about this big potential opportunity. I expect this to rapidly change over the next couple years with a much bigger percentage of enterprises getting to the state of fully deploying and operationalizing their big data projects. My high confidence is based on the level of executive support and attention that big data is receiving. See my blog for a further discussion of how executive support is fueling big data’s golden run. In some sense, this level of broad and diverse executive support is unprecedented.

From a timing perspective, there couldn’t be a more opportune time to get your big data projects initiated. I wish you the very best on your big data initiatives.

  • Haranath GnanaHaranath Gnana
    Haranath Gnana has over 20 years of experience in information systems, and is a Practice Area Leader at Saama Technologies where he was instrumental in creating the thriving Life Sciences and Healthcare practices. His technology skill set includes the development of complex information management and business intelligence systems, enabling enterprises to make data-driven informed decisions. Customers benefit from his broad knowledge of the information management lifecycle from formulating BI roadmaps and strategy, to transforming strategies to operational realities.

    During his time at Saama, he was also responsible for their "labs" initiatives to explore emerging technologies, both commercial and open source, to derive additional value for customers. As part of the labs initiatives, he built Savii (Smart Audience Viewing Intelligence & Insights, an enhanced variation of Nielsen ratings for the TV, based on TiVo data) and PSL (Predictive Store Locator, which small retailers use to identify optimal store locations) by leveraging syndicated public data sets, along with open source data mining and predictive modules. He received a patent for his pioneering work in PSL. This broad experience and expertise has helped him enable clients to adopt and leverage leading technologies such as BI appliances, mobile BI, and more recently he has been involved helping clients understand and embrace the big data phenomenon. Haranath has been published in leading outlets such as Information-Management.com, BeyeNetwork.com and Office of the CIO (oocio.com).

    Haranath completed his MBA with Honors from Lucas Graduate School of Business at San Jose State University, and his B.S. in Electronics Engineering from Bangalore University. He is a Certified Business Intelligence Professional (CBIP) in Leadership & Strategy at the Mastery Level from TDWI.

Recent articles by Haranath Gnana



Want to post a comment? Login or become a member today!

Be the first to comment!