The “Top 100 Airlines”
- as “chosen by passengers”
Why my gut feelings
(and the teachings of my Marketing Professor)
tell me not to rely the World Airline Awards Ranking
I love flying. I love airlines, aircraft and airports. If you ever met me on an airport visitors platform inhaling unhealthy smells of kerosene and tell me that this is a wasted day in life, I would smile at you and go on spending my day there.
At the same time I know that being a passenger, cabin-crew, ground-staff or cockpit crew, apron-worker, airport traffic controller or air traffic controller is far from the romance of flying of the past (sorry for leaving out the security staff of our days).
Indeed it seems hard to select the right airline and to transfer through the right airport to have at least some kind of pleasurable travel experience, one which we all keep hoping for.
SIDENOTE: Some of the lack of pleasure we all create ourselves with our own attitudes. If you board a flight grumpy, chances are you will leave the plane at least as grumpy as before. Worse: You may leave grumpy cabin-staff behind who then give the next passenger a worse treatment than you got making him feel as miserable when de-boarding as you are.
Think of this and always board with a smile on your face and in your heart!
A Rating to the Rescue?
There seems to be help along the way of travel planning: Since 1999 there are the World Airline Ratings, which according to their website are also being referred to as the Passengers Choice Awards. They bring transparency to airline service. Right?
Well I might be lousy in navigating their website but in all the years I looked at their Ratings I never found a description of their sample method. If you are in market-research you know you get toasted if you cannot present such data to clients (well at least if you were pitching or presenting to me).
Yes, they survey millions of participants each year according to their website, but who are they, where do they come from, how does the online questionnaire get promoted, where is it made available and how does it look like? And how is it being made sure that people who rate a flight from A to B actually took that flight? There is nothing said about that on their website!
And while one may refer to the questionnaire on the website as probably being one instrument with which the data is collected, I cannot see a way in which this is protected from fraudulent abuse. Indeed there had been a controversy about the ratings in 2012 and Skytrax had to change the wording of their marketing according to the linked article.
Still a look at the current online-questionnaire does not reveal which factors and at what weight do influence the final ranking of the World’s Top 100 Airlines.
And no, methodology in this case is not a secret which much be maintained to stop competition from copying the ranking. Cluster-, Regression-, or any other types of Analysis are the same all over the world. They work on Mars! But anything from the KPI measured to weights used remains covered.
Despite all these shortcomings the Ranking gets mentioned in the media and published throughout the world, impacting future choices by the travelling public everywhere! And I feel it shouldn't.
The PhD student who supervised the writing of my marketing thesis would have thrown me out (maybe even out of the window), had I shown up with such a survey description back in the days.
So even if numbers are boring (I promise to use very few) let’s check the numbers for the Top 10 Airlines in the Overall Ranking for 2015 (any flight distance, any travel-class), some runner-ups and compare these to some hard facts about the respective airline in a little bit of…
Crunching the Numbers
…which is not so easy since most data is hidden these days behind pay-walls and the three big EU airline-groups (Lufthansa, IAG, AirFranceKLM) prefer not to publish passenger numbers by brand. But the people keeping Wikipedia up-to-date provide a lot of numbers and some can be found on airline websites. So it has to be noted that the following numbers may not be consistent because of different ways in counting the number of destinations and fleet size, unclear in- or exclusion of subsidiaries, code-share agreements and other factors. Yet the Big Picture gets clear….
….and one has to come up with some questions:
1. How many ratings are there per airline, given that DELTA Air Lines (ranked 45th Globally) carries roughly 19 times as many passengers as EVA Airways (ranked 9th Globally) and how should it be possible to take network-size or advantages from operating a small fleet out of the overall equation to provide a “level field” without distorting the results which should reflect choice as a result of rating?
2. The choice of an Airline is based on destination, frequency, price and personal preferences (no order). Why does the fact, that an airline gets massively chosen to fly with, not reflect it its ranking since choice is a function of (pre-flight) rating?
3. Can you really rank airlines as a whole (in disregard of their business model and markets) in a way that it makes sense and serves both the travelling public as well as airlines in improving their competitiveness?
Airline Choice vs. Airline Ranking
Statistics has the “cool” rule were any sample above N=30 puts you on the safe side, with any of your conclusions drawn form that sample (well at least kind of). So in plain terms it does not matter if you have 50 or 50,000 people answering because the answers would (“in theory”) just not change too much on what to expect as an outcome. However if this were true in “real terms”, no one would invest in Big Data.
Having much bigger samples allow you to group data much differently and dig much deeper into all angles of view and into all aspects of the topic. The fact that all Airlines — big and small — are put into one overall basket by the World Airline Ranking seems to show that, whatever data they generate does not really go through more complex analysis. The questionnaire they have on their website for airline reviews is my best proof, since the data which can be generated with it, hardly allows for complex data operations.
And while the choice of an airline to fly with from A to B in the USA seems to be a choice between the devil and Satan — given the many complaints people voice on the web — the choice happens — among other factors — due to customer preferences.
Unless one does cite monopolies or oligopolistic market structures to be in place in US air travel (which force choices upon customers which they would not make under free market access conditions), there is no reason why airlines chosen by such large numbers of customers should rank consistently low.
United Airlines (according to the IATA World Air Transport Statistics, 59th edition) provided 143,344 million Revenue Passenger Kilometers (RPK; that is all paying passengers each multiplied with their distance flown) in international travel, making it the number two airline in the world. Next are Lufthansa, British Airways and DELTA. United is a global choice in international air-travel, which is largely a competitive market. Yet they rank outside the Top 50.
Even Southwest Airlines, usually well received by the travelling public in the US (also see table further down), ends up behind Azerbaijan Airlines, a carrier which has 31 aircraft and serves 22 destinations (compared to SWA’s 685 planes and 96 destinations). Now do not get me wrong: I have neither flown Southwest nor Azerbaijan. Maybe the latter on average does outperform Southwest in customers experience. But how likely is it that a lousy SWA would get chosen under the reign of competition?
If a ranking does not reflect de-facto choices made by customers, then either the ranking or the choices made by customers are flawed.
Airline Rating vs. Airline Choice
The World Airline Ranking seems to de-couple who people choose to fly with and how they feel with the experience of what they chose (rating). This only makes sense when the choice is not sufficiently determined by the in-flight experience. However then the Passengers Choice Award makes no sense at all, since airline choice is not significantly impacted by what passengers will get in terms of treatment and amenities (such a scenario is possible where for example price or reward-programs overwhelmingly influence the choice).
Since we have no idea (and I guess Skytrax has neither) how many of the respondents who rate airlines are frequent flyers or fly different airlines on the same route, we cannot anticipate how well-founded their expectations are. Yet the gap between what people expect upon choosing an airline and what they get is really at the core of the rating of an airline experience (or any other service experience for that matter) afterwards.
So we do not know how informed passengers are upon booking their flights or having their flights booked for them. In the latter case the choice of an airline is independent of the customer’s expectations, just as much as it is under the regime of a monopoly. Here choice cannot be related to rating.
Most interestingly we find that 13 of the Top 25 airlines in the world are from Asia and three each come from the Middle East and Oceania. So 19 of the Top 25 airlines have a substantial part of their business rooted within Asian markets (origin and destination as well as transfer). While this may hint at either high expectations being met by the airlines or low expectations being consistently exceeded (see above), one could also ask the question if the culture of respondents from these markets does lead to more modest ratings, more praise or less criticism.
While the gossip of the airline industry maintained in the past, that Europeans fly on Asian airlines while Asians fly European airlines between the regions, the fact that the World Airline Ranking does not provide any information about the cultural background of respondents per airline leaves a great question-mark behind all the ratings provided.
Airline Ranking vs. Airline Rating
Indeed ethnic aspects may play a great role in rating seat comfort, for shorter people do have fewer issues with seat-pitch than taller people or obese people (with the latter being attributes usually found with Western travelers). So the same seat-pitch may get a better rating in Asia than in Europe or the USA leading to a better ranking for the Asian carrier (or a carrier with a high number of Asian travelers) when indeed the in-flight experience provided — in terms of seat-pitch — is exactly the same!
While odds like national pride and lack of travel experience may result also in exceptional ratings for the “Homecarrier” these tend to get lost statistically due to the normalization of results through large numbers (sample N>30) as I pointed out. Still the sample size matters and there is no number of ratings given along with the ranking of each airline so we do not know how if out of those 18.89 million survey takers 100,000 or 400,000 rated ANA or Turkish Airlines.
As an indication we only have the review website where for Azerbaijan Airlines and you will find a total of THREE REVIEWS in 2014 (overall they have 36 reviews). Have fun reading them: Two of the ratings are from “different US citizens”, start with the same phrase and I never met any Americans writing/talking like that (but then I do not know all types of Americans). Southwest Airlines which ranks behind the Azerbaijanis in the Global Ranking hat 529 reviews online and already more in 2015 than the airline they trail by one rank has in total.
Overall we find that seven of the Top 10 Airlines in the World do get more reviews per passenger than other airlines.
While I have not conducted a correlation analysis between the ranking position and the reviews per 1000 passengers (since these reviews not seem to be the key source for the World Top Airline Ranking given their low numbers compared to the 18.89 million customer surveys completed by passengers over a nine months period for the 2015 survey according to Skytrax), it is of interest to note that some airlines are more successful in promoting the posting of reviews on the Skytrax website to their customers than others or that the review site is not equally known globally.
It is reasonable to assume that those who more successfully promote the filing of reviews also will be better in encouraging their passengers to fill out the major questionnaire on which the World Airline Ranking is based.
However given the low numbers of reviews we can see in the web, I find it hard to explain how the real questionnaire can attract what must be a median of ~73,000 ratings per airline over a nine months period, when Aegean, the “Best European Airline 2015” was only reviewed 324 times since November 2009.
According to SimilarWeb the monthly traffic (Desktop) never exceeded roughly 1.4 million visits in the past six months, of which 40 percent came from the US (therefore most likely rating flights beginning or ending in the USA). So 50 percent of the ratings seem to be done using mobile devices, a factor worth mentioning. However this information remains in the shadow. Those with an ALEXA Account can also dig a little bit deeper into the sites statistics and traffic numbers if they like.
Again here: The lack of transparent public information about the conducting of the survey is a major point for criticism.
A Top Airline Ranking
in this way does not work
Based on what I have written it should be clear that neither the travelling community nor the airline industry does profit from the current Passenger Choice Awards in terms of competitiveness. The link between Passenger Choice, Rating and Ranking looks broken along the line. And with no transparency in the whole undertaking it is next to impossible for an airline to benchmark itself using this ranking or for passengers to evaluate differences based on personal preferences.
Indeed, I feel little surprised that the ranking hardly passes on any other information, than what common wisdom in the industry is and can be seen on many pictures of cabin interiors published on the World Wide Web. Everybody knows that the ME3 (Emirates, Etihad, Qatar) do provide unique amenities and that Singapore Airlines always has a top cabin service and cabin experience on offer.
And we know that European airlines are trailing them. And we know about the dismal shape of the US3 (American, Delta, United) airlines, though they carry more people every year than any other airline. This size comes at the price of a greater likelihood to fail and to experience misfortunes due to more customer contact. Those then get published much faster in the Internet Age than ever before, also probably because Americans are less shy about strong words for poor service they receive. It must be noted that not all airlines in the Top 100 do operate in an equally outspoken cultural environment as US carriers do.
So in my opinion the World Airline Awards are just worth the PR that follows them and the potential bookings from a positive attitude towards an airline created by media reports — should personal preferences make the difference in airline choice, which this ranking actually seems to contradict (if not for all markets, then for a major portion of them).
And then there are the “little odd things” like Air Astana from Kazakhstan (ranked 40th Overall) suddenly seeing an influx of top ratings/reviews in April 2015 with six excellent ratings, four published on the same day of April 3rd. Such occurrences are just too improbable to happen.
The current overall ranking suggests that jetBlue should borrow from Kazakh airline Air Astana (10 ranks better) and Southwest should aim to improve by copying from Aeroflot (21 ranks better). But would that fit their business models? No statistical assumption can “equalize” caviar and burgers so that they fit into one ranking. The World Airline Ranking caters to airline thinking not customer experience. That’s a major fail.
And while the overall ranking mixes business-models, airline sizes and market-segments, the sub-categories reflect traditional categories of thought of the Airline Industry but not categories of customer attraction. Plus the subcategories are so many that 44 of the Top 100 Airlines of the world are a winner in at least one sub-category (245 airlines rated)!
Indeed, if the ranking would be meaningful the alarm bells at Lufthansa should be ringing like hell right now because Lufthansa (ranked 12th overall) has spent the larger part of 2014 handing over all flights not feeding into either Frankfurt or Munich to it low-cost arm Germanwings, which ranks only 72nd in the Global Top 100 Airline Ranking (one rank behind Germany’s second largest carrier Air Berlin). As a logical consequence of the World Airline Ranking’s findings, Germany is now the worst national market to travel within by air, ranking even behind the USA.
But my bet is on the LH management remaining calm about the findings and not for reasons of neglect or laziness and also not because the 72nd rank world-wide still means that Germanwings is the 3rd best among European Low-cost airlines. My bet is they rather focus on protecting their market share in travel within Europe against low-cost carrier Ryanair. The Irish airline has carried more international passengers than any other airline in the world according to IATA. But even with almost 87 million passengers carried in 2014, Ryanair does not show up in the Top 10 of Global or European Low Cost Airlines in the Skytrax ranking.
Rating and Ranking Airlines
has to be different
With no transparency on the sample, the measuring and the items involved in concluding a ranking position, with no figure published which relates to at least a cardinal ranking (like in the JD Powers or ACSI rankings of airlines),the whole World Top Airline Rating does not pass my personal standards for market research.
Definitely my Marketing Professor would have thrown a survey/study with so little information into the paper bin along with the ranking and the student who came up with it.
Given the lack of protection against potential fraud via the web-interface currently visible and the ability of airlines to influence at least the results of the Airline Reviews through promotion towards their happiest customers the overall results cannot be considered to be robust, even if they partially equal common industry knowledge based on airline investments into their in-flight services.
The issue of robustness is though not one arising just within the Passenger Choice awards. It is something flattering the whole Airline Industry with all Rankings for airline customer satisfaction, airline performance (in operational and service terms) and quality, producing partially very inconsistent results.
With such overall inconsistency it is hard for the Airline Industry to improve, given that all rankings have a different methodology yet claim to measure similar or equal aspects of airline performance with which the customers are confronted. Choice, as a result of competition between airlines though seems to be part or no survey’s factoring.
My Personal Conclusion
No matter if you work for the media, are just a simple traveller or someone enthusiastic about the airline industry: Please do not invest your precious time on the World Airline Award, as I have done, by putting some numbers together and writing down what has been on my mind for years when it comes to this ranking.
Do the best you can probably do when it comes to the “Passenger Choice Award”: Take my advice and ignore it, no matter what airline managers say about it. Make a choice upon your own past experiences, the resulting expectations and the realistic choices you have when going from A to B.
The world needs a better Airline Rating and Ranking
(Personal Note)
Yes, I would like to change the World of Air Travel. Who would not like to improve something as great as air travel by giving the travelling community a better voice, more fun and airlines new tools and data they can actually work with to become better?
So should you — by accident — feel the same and be what people call these days a “patient investor” (or patient angel for that matter), a programmer/coder, database specialist, an API jockey, Big Data juggler or Designer: Please get in touch with me, for I want to build a list.
While my sidekick in this field is currently resting, I have this wish to get it off the ground with a bunch of people who love the smell of kerosene and who board a flight with a smile because it makes flying more fun for everybody.
