When I went to my first big data conference almost three years ago, I thought I had been transported to a parallel universe: everyone was talking about data and analytics, yet data warehousing, SQL, and relational databases were dirty words.
Then, I looked at how people were dressed. I was the only person in the hall with a sports jacket, collared shirt, and leather shoes. Everyone else was wearing jeans, t-shirts, and sneakers and sported a pony tail. Then, it dawned on me: these were Java developers who had outgrown MySQL and were looking for a more scalable open source platform to run data-intensive, Web-based applications. And Hadoop was the answer to their big data dreams.
Immersing yourself in a foreign culture often crystallizes who you are and where you come from. For the first time in my professional life, I realized that I was a data guy from corporate IT who was wedded to commercial software and SQL-based processing. Standing brazenly in my blue blazer amidst a sea of Java coders, I also realized who I wasn't: an application developer who valued open source software.
Yet, my presence at this early big data event symbolized the beginning of the convergence of these two distinct communities: "Data people" and "applications people" have worked side by side for many years but rarely intermingled or aligned approaches. Fast forward two years. The big data conference I attended this fall had just as many "suits" as pony tails in the audience. The convergence is proceeding apace, as both communities recognize the opportunities of joining forces as well as the risks of remaining isolated.
Opportunities and Threats
Opportunities. For SQL-based vendors, the world of Hadoop and NoSQL opens new lucrative markets consisting of customers that want to harness large volumes of unstructured and semi-structured data for business gain. For Hadoop vendors, SQL-based products represent hundreds of potential applications that can legitimize the Hadoop platform once they interface with or are ported to Hadoop.
Threats. At the same time, Hadoop and NoSQL products represent a huge threat to traditional SQL-based vendors. Hadoop is like a swiss army knife that can be used to do almost anything. Consequently, many advocates believe Hadoop spells the death knoll of SQL-based databases and data warehouses. And they might have a point, since many data warehousing managers are just starting to question why they would want to move data out of Hadoop to do query, reporting, and data mining.
Conversely, SQL-based vendors, which collectively represent hundreds of billions of annual sales, aren't likely to cede this new market to a handful of open source upstarts. They are already circling the wagons, coopting Hadoop and NoSQL software by embedding them into their commercial products. This surround-and-drown strategy could spell the doom of independent, open source Hadoop vendors.
The only remaining question is which community wins in the end? My bet is on the commercial SQL vendors, which are much larger, more established, and offer robust, enterprise-caliber products that today's organizations rely on to run their businesses. They may have to radically transform their architectures and products suites to co-opt upstart Hadoop and NoSQL approaches, but they'll do what they need to stay on top and in control.
Posted February 13, 2013 1:58 PM
Permalink | No Comments |