Industry Research« Selected Data Warehouse Vendor Ratings | Main | Incredible Credibility Variances » Disecting the Magic QuadrantWhile speaking on technology portfolio management at the Wilshire/DAMA conference this week, questions about what's behind the famed "Magic Quadrant" came up. Now I certainly don't want to bash Gartner or other analyst firms outright. What they've done to keep vendors somewhat veridical and help IT organizations think through issues is admirable. It's just that their speculative, subjective method is seen increasingly as sub-optimal, particularly when pitted the collective experience of actual in-the-trenches IT professionals. It's no wonder why IT analysts can achieve rock-star status within the industry, even while playing the same tune (i.e. presenting the same old research) year after year. We're a culture and species that revels in royalty and stardom. Recently researchers at Duke University found that primates will give up treats (or "pay") to see pictures of dominant troop leaders. According to Satoshi Kanazawa, an evolutionary psychologist at London School of Economics, our weakness for endorsements traces back to our "caveman" brains. For better or worse we are hardwired to gush at celebrities (even if Natalie Portman did shave her head for her next role). But back to quadrants-n-such. Before you as an IT organization rely too heavily on analyst-generated vendor handicapping methods for selecting IT solutions, or before you as an IT vendor get too jazzed/wigged about where your dot is, ask yourself (or better yet, ask your IT analyst) the following questions: 1. When was the last time the research analyst used or even saw the product operating in the field? How much are they relying on input (e.g. surveys, marketing materials, demos) from the vendors themselves? What quantified user feedback is included? 2. How many dimensions are the vendors rated on? Two? Ten? Is this level of "analysis" really sufficient to discriminate vendors and solutions? 3. Which vendors are absent and why? Many upstart vendors who can't yet ante-up to get on an analyst firm's radar have legitimate offerings. 4. How many total IT solutions are rated in a year? The answer is likely fewer than you have in your entire IT shop. 5. How often are the charts updated? Annually? Bi-annually? Don't vendors come out with new releases (at least minor ones) and new features more often than that? 6. What do real IT pros who really use these products really think? How do they rate their performance, scalability, usability, value, etc? How do they rate the vendor's support, credibility, licensing practices, training, etc? If you'd like answers to these questions or if you don't know who once said, "The magic quadrant is the most overused, misleading, worst representation of anything," write me at doug.laney@evalubase.com to find out. You'll be quite surprised. Or not. Just telling IT like it is, Be sure to make your voice heard, get instant peer comparison scorecards, and qualify for complimentary access to all Evalubase research. Submit a confidential evaluation or two of your favorite (or least favorite) BI products today: www.evalubase.com TrackBackListed below are links to weblogs that reference Disecting the Magic Quadrant:
» Thoughts on Industry Analysts from Thought Leadership |
Comments
Unfortunately, the Magic Quadrant has nearly become a standard in our lingo - which is almost a pre-requisite as level entry pass to discussions with peers, prospects, competitors, press, etc.
Yet, I concur with Doug that new market analysis and assessment approaches are now meeting better the needs of small and large corporations - by offering more pertinent content from knowledge workers using hardware & software products, real-time data from current projects and more detailed/accurate information, which is really what one needs to get their project going and off the ground.
As I have been working for large software and hardware vendors, it appears that customers still demand our positioning in the Magic Quadrant yet, when it comes to real-life implementation, other resources are required.
So, this is not a perfect world but one where there is a need for it all (Gartner's magic quadrant for pre-sales activities and other analysts for pre & post sales activities - including the most important part: implementation!).
Just telling it as I experience it from the Silicon Valley and traveling around the world!
Anyck Turgeon
Posted by: Anyck Turgeon | May 28, 2005 9:48 AM
Doug's comments are right on the money...from the misguided status and weight that is associated with an analyst and his/her opinion to the manner in which those opinons are correlated. I suppose that there was a period of time when these shallow analyses were important, but with technologies that provide (and promote) collection from the masses that actually use these products, one must question the validity of antiques like the "magic quadrant."
In fact, another couple of questions that I would add to Doug's "spot on" list are:
7) What is the historical accuracy of these analysts predictions regarding the products in their space?
8) What is the basis for analyzing post-sales implementations and total cost of ownership?
In addition, I would further qualify Question #2 by asking: What are the details to the dimensions that are analyzed? In other words, how do I know that products are fairly analyzed with consideration to their inherent differences?
Thanks Doug, for sharing your insighful comments.
Posted by: Ken Kane | May 30, 2005 9:28 AM
I too agree with Doug 100%.
One area that is changing for the better is the identification of all vendors. With more vendors providing smart directories, at least the identification of all the vendors by product / service type should be easier.
Information vendors that provide strategically relevant industry information and trends, including a list of all competitors, but without any subjective product analysis, will have no bias or reason not to include all vendors.
IT organizations can then better ask Gartner and other analysts firms why the additional firms were not included in their analysis. Alternatively, the IT organization can at least do some basic research themselves. Smart directories will also help Gartner and others because they will serve as another check for them to make sure they don’t accidentally or unknowingly skip some vendors. (We cover approximately 2,000 non-overlapping IT and business consulting industries, and with global start-ups entering the market all the time, we, like Gartner, will always miss some – but with a good feedback system – the missing usually become the found very quickly.)
Posted by: Alan S Michaels | September 3, 2007 7:43 PM