Industry Research

« April 2005 | Main | June 2005 »

May 26, 2005

Disecting the Magic Quadrant

While speaking on technology portfolio management at the Wilshire/DAMA conference this week, questions about what's behind the famed "Magic Quadrant" came up. Now I certainly don't want to bash Gartner or other analyst firms outright. What they've done to keep vendors somewhat veridical and help IT organizations think through issues is admirable. It's just that their speculative, subjective method is seen increasingly as sub-optimal, particularly when pitted the collective experience of actual in-the-trenches IT professionals.

It's no wonder why IT analysts can achieve rock-star status within the industry, even while playing the same tune (i.e. presenting the same old research) year after year. We're a culture and species that revels in royalty and stardom. Recently researchers at Duke University found that primates will give up treats (or "pay") to see pictures of dominant troop leaders. According to Satoshi Kanazawa, an evolutionary psychologist at London School of Economics, our weakness for endorsements traces back to our "caveman" brains. For better or worse we are hardwired to gush at celebrities (even if Natalie Portman did shave her head for her next role).

But back to quadrants-n-such.

Before you as an IT organization rely too heavily on analyst-generated vendor handicapping methods for selecting IT solutions, or before you as an IT vendor get too jazzed/wigged about where your dot is, ask yourself (or better yet, ask your IT analyst) the following questions:

1. When was the last time the research analyst used or even saw the product operating in the field? How much are they relying on input (e.g. surveys, marketing materials, demos) from the vendors themselves? What quantified user feedback is included?

2. How many dimensions are the vendors rated on? Two? Ten? Is this level of "analysis" really sufficient to discriminate vendors and solutions?

3. Which vendors are absent and why? Many upstart vendors who can't yet ante-up to get on an analyst firm's radar have legitimate offerings.

4. How many total IT solutions are rated in a year? The answer is likely fewer than you have in your entire IT shop.

5. How often are the charts updated? Annually? Bi-annually? Don't vendors come out with new releases (at least minor ones) and new features more often than that?

6. What do real IT pros who really use these products really think? How do they rate their performance, scalability, usability, value, etc? How do they rate the vendor's support, credibility, licensing practices, training, etc?

If you'd like answers to these questions or if you don't know who once said, "The magic quadrant is the most overused, misleading, worst representation of anything," write me at doug.laney@evalubase.com to find out. You'll be quite surprised. Or not.

Just telling IT like it is,
Doug

Be sure to make your voice heard, get instant peer comparison scorecards, and qualify for complimentary access to all Evalubase research. Submit a confidential evaluation or two of your favorite (or least favorite) BI products today: www.evalubase.com

Share:

May 16, 2005

Selected Data Warehouse Vendor Ratings

As I have cautioned several times, Evalubase's research method comprises an ongoing study of the enterprise technology market. While our method is imminently more empirical (and far less magical) than "quadrant" type charts and doesn't include any vendor supplied data whatsoever, some of our data is quite preliminary. We caution vendors in particular about spouting any of these findings without checking with us first.

That disclaimer out of the way, here are current overall technology and vendor ratings for solutions/vendors predominant in the data warehouse market. Vendors offering business intelligence, data integration, data administration/modeling, data quality and metadata management solutions were selected for this comparison.

Overall technology ratings comprise user-supplied ratings of functionality, performance, scalability, maintainability, usability, security, portability, ease of integration, ease of implementation, satisfaction and value. Overall vendor ratings comprise user scores of the vendor's credibility, responsiveness, ingenuity, support, vitality, sales process, marketing, legal & accounting functions, licensing practices, and services & training capabilities.

At present, Actuate, Embarcadero, Ascential, Business Objects, Data Advantage Group, Firstlogic and SAS enjoy an early overall lead. Following them are Computer Associates, MelissaData, Siebel, DataFlux, i-Way, Information Builders, Informatica, Microstrategy and Cognos.

View Evalubase Chart

Many of the current scores are actually pretty close. We'll see what happens over time as you and your colleagues participate in the ongoing study. In time, we'll compare particular attributes of key competitors and study how (for example) their customer support ratings and ROI numbers evolve over time.

See you at the TDWI and Wilshire/DAMA events this month!

Just telling IT like it is,
Doug

Be sure to make your voice heard, get instant peer comparison scorecards, and qualify for complimentary access to all Evalubase research. Submit a confidential evaluation or two of your favorite (or least favorite) BI products today: www.evalubase.com

Share:

Data Management Buyers Want Performance and Maintainability First and Foremost

When asked about which factors were most important in selecting one data management solution over another, users are indicating that performance and functionality characteristics of candidate solutions were paramount. View Evalubase chart

With data volumes, velocity, and variety continuing to grow at a staggering pace, it’s no wonder why buyers of data integration and database products, for example, currently rate these factors most significant.

Following performance and functionality in importance were stability, cost-of-ownership, and ease of use. Least important in the decision are scalability, ease-of-integration, and vendor reputation.

Conversely, those involved in selecting and purchasing business intelligence tools have paramount concerns about tools' abilities to integrate with other solutions. View Evalubase chart

Next most important to BI buyers is product stability and cost of ownership. Least important are scalability and vendor reputation. So why aren't BI buyers concerned (relatively speaking) with scalability? These solutions are already proven to be scalable? Web-based implementations are naturally scalable? Probably a little of each.

See you at the upcoming TDWI and and Wilshire/DAMA events this month!

Just telling IT like it is,
Doug

Be sure to make your voice heard, get instant peer comparison scorecards, and qualify for complimentary access to all Evalubase research. Submit a confidential evaluation or two of your favorite (or least favorite) BI products today: www.evalubase.com

Share: