Business Intelligence Network - The Vision for BI and Beyond

Selecting the Best BI Tool

Published: April 3, 2007

This article highlights eight steps toward selecting the best business intelligence tool for your company.

The business intelligence (BI) market has changed considerably in the last three years: BI pure-play vendors have broadened and integrated their solution sets, and enterprise resource planning (ERP) and relational database management system (RDMBS) vendors have stepped up their pursuit of the BI market.  

The good news for customers is better products, increasingly able to meet a broad range of user needs, with solutions that scale across the enterprise. Mixing and matching best-of-breed BI modules from multiple vendors is no longer necessary. The bad news for customers is that what once were obvious vendor and product differences are now less clear; some vendors are pursuing similar strategies, and while product differences still exist, they are much less apparent. 

Given these changes, customers should plan for the following:

• Selections and standardizations may be made based more on strategic considerations, with the assumption that existing product capabilities are “good enough.”
  
  
• Selections and standardizations will take longer, as customers must spend more time evaluating and understanding product differences. Ultimately, certain specific product differences may tip the scale toward one vendor, or when there is a “tie” in best fit, strategic differences will be the deciding criteria.

With either approach, it’s critical that customers understand the role of the BI platform in delivering measurable business value. It is the face of BI.  If you underestimate its importance to engage users and facilitate fact-based decision making, your BI project will have mediocre success. Conversely, technology issues should not be allowed to overshadow the business objectives of the BI initiative. No matter which solution you select for an enterprise standard or new BI implementation, naysayers will second-guess that decision. The key to managing such second-guessers is to follow an objective, agreed-upon methodology.

This article highlights eight steps toward selecting the best BI tool for your company.

1. Form the BI selection committee: The selection committee should be comprised of a cross-section of stakeholders from different functional areas and user segments. This includes IT report developers, data warehouse modelers, power users and information consumers (report consumers who may or may not log into the BI tool but who need the information for decision making). Current BI application owners are prime candidates for serving on the core selection committee as they have unique insights on current successes and unmet needs. Giving them ownership in the selection process minimizes the risk of their second-guessing you later. At the same time, keep the selection committee small enough to be effective. The selection committee will elicit feedback from a larger user constituency to ensure buy-in. A number of companies skip this process entirely and make BI purchase decisions at the departmental level. While this may work for point solutions, companies that take an enterprise view of business intelligence report greater success rates. 

2. Define target users and usage scenarios: Despite BI’s maturity, as an industry we do a poor job of understanding different user profiles and correspondingly matching product capabilities. In the overall BI life cycle, the focus is too much on creating a data mart or a report, rather than on who will interact with that report and how.  

Different user types require different tools or interfaces. This is not to say that you cannot buy a complete solution from one vendor: increasingly, BI vendors offer a full spectrum of products including production reporting, business query, dashboards, OLAP, and predictive analytics. But as you define your user profiles and use cases, you may discover that one group has unique needs with specific functional requirements. Understanding these user segments is critical in managing the scope of your selection and resolving conflicting requirements.

3. Refine information requirements: In our quest to see these BI tools with all their sex appeal, refining the information requirements is the most overlooked and least understood. For companies that have multiple BI tools, they know the harsh reality that every tool handles data and schemas slightly differently. Unless you simply want another pretty report with “bad” data, you have to incorporate information requirements into your BI selection process. This step is different from defining the data elements and data sources to build the data warehouse, and instead, considers how the data will be analyzed. For example: users may express the requirement to view sales with inventory to calculate Days Sales Inventory by various product groupings and time periods. This single requirement translates into a host of technical features such as:

• Multipass SQL to query two fact tables in a data warehouse or multiple measures in an OLAP database,
  
  
• Semi-additive measures to aggregate inventory across product groupings but not across time periods, and
  
  
• Automatic aggregation of individual rows of data to view totals for the year or product group.  

I’d venture to say that all the major BI tools can handle this type of business requirement, but they do handle them in drastically different ways, with varying degrees of ease, and by leveraging different components in the BI architecture.

The selection committee must understand these differences and know which approach fits within the desired architecture and organizational abilities. 

4. Define and rank selection criteria: This is the juicy bit of the selection process and one that customers and vendors meet with a fair degree of trepidation. There are multiple methods to capturing user requirements: individual user interviews, gap analysis, and brainstorming sessions, to name a few. Key, though, is translating a requirement into a BI tool capability. For example, users will rarely say, “We want a BI tool with a business metadata layer.” However, they may say something like, “We want to create our own reports without having to know SQL.” The feature of a business metadata layer fulfills this requirement. In order to develop a list of requirements, you need to know what’s possible and you need to consider emerging capabilities. In this regard, developing and ranking your selection criteria is an iterative process. 

Product features are often easier to rank than strategic considerations. Agreeing on the relative importance of something like “BI market leadership” versus a capability like “real-time spreadsheet integration” is a difficult and often contentious task. Ideally, the same vendor will score high in both strategic considerations and product capabilities, but currently that’s not the case.  

To get you started, BIScorecard.com contains a complementary list of often-cited requirements. Use this as a review for checking the completeness of your requirement list.  

5. Request for information (RFI): It would seem like a glaring omission to skip this step, yet increasingly, I question the value of it. RFIs create a lot of work for the vendor and not much value for the customer. Of recent RFI responses I’ve reviewed, there is an increasing tendency for vendors to say “yes” to each requirement, even when a better answer is “not really, but possible with lots of workarounds.” To be fair, some vendors are more honest than others and requirements are subject to interpretation.  

Some requirements are show-stoppers that an RFI can help weed out. For example, if your company’s standard operating system is Linux and the vendor doesn’t support Linux, that requirement might cause the vendor to be removed from your short-list. Other requirements are not so clear-cut. To improve the value of an RFI, define your requirements well to avoid misunderstandings between you and the vendor. Second, ask for specific product names, feature names, and explanations for how the requirement is fulfilled. Know when the answer requires a yes/no response. Third, keep the RFI short, emphasizing the critical requirements that will be decisive in your selection or standardization. While I consider the BIScorecard checklist almost too detailed at 100+ requirements, customers constantly send me lists with close to 200 requirements. With that many requirements, both the selection committee and vendor can quickly drown in the details, rendering the requirement list ineffective.

Finally, complement vendor RFI responses with a heavy dose of your own research from customer references, discussion groups, and published product reviews.

6. Vendor scripted demos: Vendor demos should be scripted so participants can objectively compare vendors and products. Prepare a consistent agenda for each vendor to follow. In the agenda, allow time for a discussion of strategic considerations as well as specific product capabilities identified as critical. Be sure to invite an extended user base to the demos so you can elicit qualitative feedback and ensure users have a stake in the decision-making process. Based on your priorities defined in step 4, ask demo participants to score the vendors on their ability to meet the various requirements. Scripted demos can either be based on the vendor’s sample data or based on your own. Using internal corporate data further identifies how each vendor’s tool is different, yet it requires a large investment from both you and the vendor. Such an investment is advisable with a handful of vendors but is impractical with many vendors.  

7. Determine best fit: Determine best fit using your requirements matrix defined in step 4, score the RFI responses and demo feedback. Incorporate strategic considerations, qualitative research, and customer feedback to determine which vendor(s) most closely matches your company’s short and long-term BI needs. If you have one clear leader, do not fully dismiss the runners-up. You may find out during the proof of concept, contract negotiations, or pilot that your first choice has insurmountable issues.

8. Proof of concept: You may only have one or two vendors that move onto the proof of concept stage. The proof of concept stage is your chance to test the tool in your environment. It is only a test, though. At this point, it’s important to keep the selection committee focused on the critical requirements rather than endlessly playing with the software or attempting to create usable reports. The proof of concept may be a throwaway: its sole purpose is to confirm that the product works as you expect it to. Carefully manage the scope to use one subject area and a handful of sample reports for the proof of concept. The sample reports should be based on information requirements defined in step 3 and be of moderate complexity (don’t waste your time with simple list reports that all BI tools can handle, or, conversely, the killer report that took your best programmer a full month to design). The proof of concept will give you insight into how you may need to adapt the rest of your BI architecture, but it is not the point at which you solve all the implementation problems.  

Oh, and did I mention that these eight steps only get you to a BI product decision? You then of course have to negotiate a contract, implement, deploy, leverage, and promote the tool!  

The BI market has changed significantly in the last few years. Emerging technology, vendor innovation, and acquisition have changed the BI landscape dramatically. Recognize the importance of selecting the right vendor and solution to the overall success of your BI deployment. Ultimately, the more important question is not which tool, but rather: how will you use business intelligence to provide business value?

If you found this article helpful and would like to receive the latest insights each week from the experts featured on the Business Intelligence Network, please subscribe to the Business Intelligence Network Newsletter.