We use cookies and other similar technologies (Cookies) to enhance your experience and to provide you with relevant content and ads. By using our website, you are agreeing to the use of Cookies. You can change your settings at any time. Cookie Policy.

Data Integration for Big Data Analytics: A Q&A Spotlight with John Whittaker of Quest Software

Originally published September 4, 2012

This BeyeNETWORK spotlight features Ron Powell’s interview with John Whittaker, Director of Product Marketing, Database Management at Quest Software. John and Ron talk about why successful big data analytics requires integrating data from many sources, including NoSQL unstructured data environments. They discuss Quest’s solution that meets the access needs of  business while maintaining the governance and security IT requires.
Our readers may be familiar with Quest, but not necessarily relating to business intelligence software. Could you tell us how Quest fits into BI and analytics?

John Whittaker:
Quest Software is known in the systems management and productivity tools world, and particularly within the database world, for its TOAD product, a tool that is well received and loved by millions of users worldwide to manage and develop relational databases, such as Oracle, SQL Server, DB2, as well as new NoSQL databases. It’s from this perspective that we’re now branching into the business intelligence world. From the very beginning, our customers have played a major part in the direction we’ve taken our software development and the businesses we’ve moved into. They’ve brought us from being a development tool into the database administration world.  When users began asking for more powerful administration capabilities, we began building tools for DBAs. In recent years, we started seeing users utilizing our tools to do database provisioning as an analyst function so we built a tool for those analysts. That group of users – the analysts – is really the reason we’re entering this market space. It’s a user base that’s very technical, but that deals with a real business problem – being able to capture data from multiple sources and produce views so data consumers can take more informed actions in their decision-making process.

It certainly seems like a natural connection to go from the data to being able to visualize it through BI. Let’s talk a little bit about TOAD, Quest’s database developer productivity tool. What have you learned from your users and how has their feedback affected the development of your products?

John Whittaker: Through a constant series of feedback that we receive from our users, we build new features and we create new products. One of the areas that kept surfacing from the users who were using our analyst tool was that they consistently have a conflict with the business users – or if they’re on the business side, they have the conflict with IT. IT owns the data. They keep it secure and available, but they may not necessarily know exactly what the business users want when they ask for something from a reporting or dashboard perspective. Oftentimes, the business users themselves may not know exactly what they want. They may need to do some data exploration or data discovery to really get to the answers that they’re looking for. This desire for something that would allow users to provision data and then to consume data and collaborate was really what brought us into the BI space. We built a tool called TOAD Data Point that is purpose-built for data provisioners. We built another tool for data consumers called the TOAD Decision Point.

Marrying these two tools together is a server called TOAD Intelligence Central that allows the data provisioning persona and the data consuming persona to collaborate on different views of data. Underlying this suite of products is an architecture that has a virtual data layer or logical data warehouse technology that allows the system to be able to attach to whatever data source a user might desire.

Within the BI world, you have the major monolithic players, and their specialty is handling massive numbers of transactions, massive amounts of data and serving up dashboards to a very large number of users, but by design they may have only a subset of the potential data that exists. In any environment, you always have change. You always have new databases being added. There are always new sources of data that you’d like to pull in. With these monolithic systems, you get scale, but they’re not very agile or flexible.

Outside of the monolithic BI programs such as BusinessObjects, OBIE or Cognos, you usually have several other data sources that exist in the environment. They could be running on relational databases like Oracle, SQL Server or IBM’s DB2, or they may reside on NoSQL big data platforms like Hadoop or Mongo DB or Cassandra. They might exist in the cloud. We have many users who use Salesforce.com and would love to be able to access that data source as if it were just another available piece of database architecture in their environment. And then, of course, you have a variety of different Excel spreadsheets that exist and get shared throughout the enterprise. This reality of diverse data sources makes it difficult for IT to report against, and can lead to serious control and data governance issues when business users try to build their own data marts to do reporting. Our tool solves the problems business users face because you can access all these data points, and do it in a manner that IT is very comfortable with because we don’t break their governance rules and we apply the same security within their architecture so users can’t see anything outside the level of their user access.

It sounds like the TOAD Business Intelligence Suite really does help IT. At the same time, it gives the BI director the ability to work with IT in a more cooperative manner.

John Whittaker: Exactly. It just snaps into whatever infrastructure you have in place. The more complex and heterogeneous it is, the greater the value proposition, and we’re just a complementary technology to whatever you have, allowing for faster time to value and enabling the organization to bridge the gap between the data that exists in their siloed BI arena and the other data the business users may desire to attach to that.

John, there is a new layer in the TOAD Intelligence Suite where you connect, consume and translate information across all these distributed platforms. How does it work? What did you see happening with your users that led you to believe that this was the best tool to solve their pain points?

John Whittaker: That came out of an R&D project that we began back in 2008 when Guy Harrison, our VP of Research and Development, recognized that this “big data” wave was coming and the technologies that were beginning to be developed at that inflection point in the market were going to be something that our users were going to be very keenly interested in. And really one of the major problems that exists today with the big data technologies is the fact that there is no easy way to access these NoSQL unstructured data environments. He began building a product that we call TOAD for Cloud Databases that allows SQL querying on NoSQL databases so that a developer or DBA who wants to query a NoSQL database can write standard ANSI SQL. Guy built a universal translator technology that takes that standard ANSI SQL and converts it into whatever the native querying would be for each of the major big data environments.

That underlying architecture is really the core of the TOAD BI Suite, and it allows for an individual to write standard SQL or, in the case of the tool that we built for the business users, to drag and drop query without writing SQL, hiding that complexity from the user. It’s the underlying engine that creates this logical data warehouse in memory based on whatever views or joins you’re doing. It pulls them into a view, which can remain perpetually or can be updated on a regular basis. It’s that architecture that is the hub of the TOAD BI Suite.

Can you explain to our readers why data integration is such a necessary component in any conversation regarding big data?

John Whittaker: One of the big aspects that people are trying to get a handle right now, and one of our major uses, is big data analytics. Once you get beyond the Google or Facebook use cases and start talking about how the rest of us will use big data, it is going to be analytics. Whenever you’re doing analytics, you want to marry information from different sources. You might want to be able to correlate what’s happening within your ERP and the operational data that might exist there, with experiential data from your website. The operational data often tends to reside in relational databases, but when you’re talking about experiential data, about how people are utilizing your website or what people are saying on social media about you, that sort of data resides within the unstructured big data world of Hadoop. It’s really about being able to marry these sources together into one environment and drive better decisions based on the information there is the primary value
that the big data environment is going to provide for the normal enterprise.

How does self-service BI impact data integration efforts? What are the challenges?

John Whittaker: We really have a mantra that we’ve been sharing around the office since we started this project. It is that you can’t really have self-service business intelligence without self-service data integration. When you look at the technologies that exist today that allow you to do self-service business intelligence – beautiful renderings and visualizations and all these great powerful ways to look at data – they still require somebody to pull data from different sources to run that visualization against them. This aspect of the self-service BI market really has caused more problems for IT and increased the likelihood of data marts within the environment.

We feel, and this definitely has been true with many of the customers that we’ve worked with as we rolled out this solution, that if you can empower the provisioners and the consumers with the tools that will allow them to work together, you can eliminate the conflict or friction point between the two groups and ultimately deliver on the promise of BI, which is driving informed urgent action. That’s really what we want to see happen, but if you don’t have the data integration component in place or you don’t make it very easy for data integration to occur for the collaboration between the user who wants to consume the data and the provisioner who wants to provide the views, you’re going to have long delays, significant investment requirements in platforms that may or may not meet the needs of the user, and a perpetuation and worsening of the conflict that has existed in this environment between these two groups. You can’t really have self-service BI without self-service DI.

That makes a lot of sense. In fact, when you put it that way, it’s almost an obvious point. How do you see Quest’s solution handling the challenges of big data?

John Whittaker:
Well, it’s going to be an interesting problem that everybody is grappling with. I think we have a significant lead in this space from where some of our competitors are today since we did start our R&D cycle very early on. At the end of the day though, our greatest advantage is the relationship we have with our users and the fact that they are so vocal on our social media and our community site – TOADWorld.com, where they’re constantly bringing up things they’d like to see, voting up or voting down the features that we’re building into the next generation of product. It’s really from that collaboration with our customers that we consistently bring solutions to bear that drive down the cost of labor for organizations and increase the value that users can deliver. We really do build great productivity tools. We surround the platform with solutions that make the lives of the people who have to deal with the platform easier. We feel that same sort of zeal around making sure that we listen to our customers and solve their problems. That will be what helps us with the big data space just as it has in the relational database space for so many years.

Well, if you can create the foundation from the data side, obviously it helps from the BI side.

John Whittaker: Certainly. We definitely have a great relationship with IT, with the DBAs and the people who own the databases, and we look forward to working with business users as well, helping them experience the great enhancements that TOAD provides.

John, thanks for providing this insight into Quest and your commitment to giving organizations the tools they need for big data analytics.

  • Ron PowellRon Powell
    Ron is an independent analyst, consultant and editorial expert with extensive knowledge and experience in business intelligence, big data, analytics and data warehousing. Currently president of Powell Interactive Media, which specializes in consulting and podcast services, he is also Executive Producer of The World Transformed Fast Forward series. In 2004, Ron founded the BeyeNETWORK, which was acquired by Tech Target in 2010.  Prior to the founding of the BeyeNETWORK, Ron was cofounder, publisher and editorial director of DM Review (now Information Management). He maintains an expert channel and blog on the BeyeNETWORK and may be contacted by email at rpowell@powellinteractivemedia.com. 

    More articles and Ron's blog can be found in his BeyeNETWORK expert channel.

Recent articles by Ron Powell



Want to post a comment? Login or become a member today!

Be the first to comment!