The Truth About “Best Of” IT Vendor Lists

Theorem
Theorem
Mar 29 · 5 min read

To grow your business in today’s modern world, you need digital solutions. When it’s time to bring in a firm to solve a problem that requires technology and innovation, you want the right captain for the ship.

The internet provides a sea of possibilities when it comes to searching for a partner, but it’s all too easy to lose your bearings. The trouble? An overabundance of players in the market. IT service companies can be found all over the world, at different price points, ranging from freelancers to big, faceless corporations. How do you even begin your search, much less determine which company is the best fit for your project?

Say you’re surfing the web for a software development firm, and you come across a website that boasts: Top Lists in the Software Industry. “Great,” you think, “My search is over!”

Not so fast.

At first glance, these business review sites — such as Gartner, Forrester, IDC, and Ovum — appear to have all the answers. They lay out the “best in the business” in a neat little package to make your selection process easier. Major global corporations trust companies like Gartner for insights and analysis across a wide range of market sectors. But the reality of these lists is that they are, as one Diginomica writer puts it, “fiction masquerading as fact.”

Questionable Methodology

The issue lies with the business model. While these groups claim that their rankings are based on peer reviews, there is a lack of transparency and disclosure about how the ratings and reviews for the vendors they promote are really derived. For example, Gartner’s market research reports, known as Magic Quadrants, claim to provide a “culmination of research in a specific market, giving you a wide-angle view of the relative positions of the market’s competitors.” But when you look closer at the methodology, the view is much narrower than it appears.

As an article in Tech News World explains, “In addition to being inherently subjective, Gartner requires robotic process automation (RPA) vendors to meet stringent specific criteria, including revenue, operating budget, and client number thresholds, for inclusion and even consideration in a Magic Quadrant. If a vendor misses even one guideline, they are excluded. The problem with this is that these aren’t the same criteria that companies use to evaluate technology. Many businesses don’t care about a vendor’s size or revenue numbers per se. They care about the quality, innovation, and applicability of products, the quality of customer support and professional services, and what real customers have to say about the product.”

Gartner’s methodology has also failed to adapt to evolving technologies and markets. While other companies are producing vendor reports that evaluate process discovery and mining in the context of automation, Gartner’s continued use of antiquated methods further put the relevancy of its lists into question. As Tech News World points out, “The value of technology should be determined by its innovation and contributions, not numbers and arbitrarily assigned thresholds.

Follow the Paper Trail

Companies like Gartner and Forrester have also drawn criticism for how they generate revenue. Software development companies who want to be included on these lists must spend money to improve their position. Commonly known as “pay-to-play,” this model favors bigger vendors with lots of resources and creates major conflicts of interest for Gartner. It’s no wonder the same tech giants of the world continue to be featured over and over, while smaller or newer firms show up lower on the list — or not all.

Gartner has actually been sued twice over its “proprietary” methodology and biased financial model. In 2009, software vendor ZL Technologies filed a federal lawsuit claiming that Gartner’s Magic Quadrants were misleading and favored large vendors. In 2014, technology provider NetScout Systems filed a suit for “corporate defamation arising out of Gartner’s information technology research business practices.”

Gartner’s latest offering, Peer Insights, further widens the divide. Peer Insights is described as a hub “for ratings and reviews of enterprise technology solutions by end-user professionals for end-user professionals.” But it’s really just encouraging technology vendors to incentivize customer reviews in an attempt to gain better standing on Gartner’s lists — which is much easier to do for companies with deeper pockets (and pre-existing relationships with the organization.)

This dynamic creates a skewed marketplace for those trying to find high-quality, creative development partners that fall outside the list of typical players.

Don’t Miss Out on Hidden Gems

The companies that produce these lists certainly don’t want you to think that the money they take in has any effect on their rankings. Their argument is simple: we’re popular because we’re unbiased; we’re unbiased because we’re popular. But this circular logic just doesn’t add up. As Brightwork Research & Analysis points out, Gartner’s argument is like saying, “If McDonald’s did not offer healthy food, no one would buy from them.” And while Forrester may be second behind Gartner, they use the same techniques to influence unsuspecting shoppers into picking whichever vendor pays more.

Confused yet? You should be! These pay-to-play lists not only alter the marketplace, they make finding a trusted partner extremely challenging. Just because a software development company is included on a “Best Of” list doesn’t make it the best choice for you — it just means that it paid to be there. The truth is that these lists are simply a marketing tool for the companies who can afford them. There are tons of fantastic small to medium-sized technology firms that could be the perfect solution to your unique problem or opportunity. In fact, companies NOT listed are probably more affordable and deliver better quality products than the big box businesses!

You may get more bang for your buck by going with a smaller vendor. These companies put their money toward development and innovation rather than worrying about lists and rankings. Their work and commitment to their clients stands for itself.

Instead of putting full faith in lists from Gartner or Forrester, look for a partner that’s focused on:

  • Listening instead of pitching
  • Walking with you through the discovery process
  • Having the flexibility to meet your unique needs
  • Thinking creatively and pursuing out-of-the-box ideas

Word gets around when exceptional work is completed on time and on budget. Get recommendations, check references, and read case studies. You’re sure to find the right software development partner for your needs.

Theorem is one company you won’t find on those lists. We’d rather put our energy toward creating stellar solutions for our clients than buying into the pay-for-play games.

Theorem

We are an engineering and innovation firm.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store