One of the most buzzed-about topics of 2013, Big Data, gets a lot more interesting when we see how it intersects with the SQL arena. Some DBAs may see the rise of big data like Captain Billy Tyne of the Andrea Gail saw one of the big waves in The Perfect Storm—a black wall of sheer petabytes about to engulf everything in sight.
But as we’re seeing it play out, the worlds of Big Data and SQL are highly interdependent, creating a realm of possibility for SQL experts to add big value to their organizations. This explosion of data is also an explosion of opportunity.
Facebook’s upcoming open-source Presto release—which will enable real-time queries with an SQL interface into Facebook’s Hadoop data—is just one example. Major Big Data-SQL initiatives are on the ground or in the works at Amazon, HortonWorks, IBM and EMC. Wherever Big Data rears its head, SQL has proved an adaptable workhorse.
Even the term “Big Data” itself can be a bit misleading, conjuring the “perfect storm” of massive data. But as Thomas H. Davenport and Jill Dyché noted in their “Big Data in Big Companies” white paper published for the International Institute for Analytics, the game is as much about variety of data as it is about sheer mass: “The most important goal and potential reward of Big Data initiatives is the ability to analyze diverse data sources and new data types, not managing very large data sets.”
Big data will be driving innovation and cost reduction in a wide variety of sectors, allowing companies to integrate data and act on market forces and new opportunities. It will support existing strategies with new capabilities (for example, enabling a large retailer to execute real-time, agile pricing) and give rise to new service offerings as multidisciplinary teams learn to operationalize new insights.
And the emergence of Big Data will be far from a rip-and-replace endeavor, as Big Data solutions are integrated with legacy systems to support the unique goals of each company. For the foreseeable future, the wild new world of Big Data and business intelligence will be closely wed to tried-and-true data management practices. For companies seeking to leverage Big Data for competitive advantage, SQL experts can lend value to every piece of the puzzle—from data storage and infrastructure to how the data is manipulated, viewed, presented, consumed and, ultimately, acted upon.
Companies will have a wide spectrum of options as they decide which problems they’ll try to solve with big data and how to align their people, processes and technology to reach their goals. DBAs can help solve storage and backup issues, as well as bridge the gap between manipulating the data and making sense of it. They’ll also be counted on to reduce the pain of moving needed data among many function-specific databases—including Big Data solutions such as Netezza or Hadoop.
Has your company decided on which "challenges" to address first in regards to big data? Are you, the DBA, involved in the decision making and what do you see as the opportunities related to big data?