Venting out the vanity metrics

As a data scientist working closely with the product team in my company a lot of my work revolves around metrics and numbers. I spend a good chunk of time every week curating metrics which describes an underlying behaviour or phenomenon clearly. It is easy to pull out a lot of metrics but it is difficult to figure out the metrics that actually matter. It is easy to get lost in a bunch of vanity metrics than actually looking at metrics that are actionable and gives a good idea about the thing that we are actually trying to measure or improve.

Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies. These metrics are exciting to point to if you want to appear to be improving, but they often aren’t actionable and aren’t related to anything you can control or repeat in a meaningful way.

I am no pro in fending off vanity metrics or curating metrics that are as actionable as Arnold. But having curated so many metrics and looked at many numbers for the past 5 plus years, I think now I have a better understanding of the numbers that actually matter, compared to the times when I was making rockets out of my resumes and making it fly from the rooftop of my house hoping that it will land in some MNC. With so much data being generated in our day-to-day lives and everyone wanting their agenda or opinion to be backed by some data, we get bullied with numbers and metrics in one form or the other starting from a needle advertisement to news channels. This blog is me looking back at my life and recalling some instances and events that are fresh in my memory involving numbers and metrics which got me carried away back then but looking at it now retrospectively makes me realise that most of them were nothing but just vanity.

I was born and bought up in a Coimbatore, Tamilnadu. People used to call it Manchester of South India, owing to having a number of largescale cotton mills. Most of them are no more operational and Tiruppur which commands a Lion’s share of cotton mills which used to be a part of Coimbatore was made into a separate district. Still, some people do refer to it as Manchester of South India. Not even Manchester of India because Mumbai is the cotton King. Not sure what is the desperation behind the idea to somehow connect it with Manchester. With Tiruppur being separated, if people still want to make a connection with Manchester then Coimbatore should be called ‘Manchester of westernmost Central Tamilnadu’.

People do call it an educational hub too owing to a number of high profile colleges and schools. In fact, there really are a lot of colleges and schools. Whether they are high profile or not is debatable. When I had to switch schools during my 8th Grade my parents were looking at nearby schooling options and one metric they considered to shortlist a school was whether that school produced cent percent result in the latest high school (Plus 2) board exam. That is, whether everyone who appeared for the board exams that year from that school has passed. Even the schools in my area proudly let a word out about this metric and actually faced a lot of heat if the number wasn’t 100% because that would put the school in a bad light. What a shallow idea it had to be, to judge a whole school by having a binary flag of whether everyone passed or not. That one guy or girl who flunked could have got a fever or maybe something unfortunate happened in his life or genuinely he did not do well in his exams but what can a school do? But still, schools felt bad. The schools did their part in making this metric shine. I realised that only when I passed my 10th exams. Even to continue the eduction in the same school for plus 1 and plus 2 we had an admission process and that’s where the schools began to polish their metric. They only admitted people whom they thought would perform well in +2 boards so that they had the best chance to hit cent percent results. Its football equivalent of only taking penalties in which there aren’t any keepers. This is the first encounter in my life where I saw a metric being abused. And there is no better example of Goodhart’s law which states

When a measure becomes a target, it ceases to be a good measure.

Coming to Tuitions. Tuitions were and are still a big thing in my city. My area is clustered with tuition centres. If you were appearing for board exams and were not enrolled for tuitions in Maths, Physics and Chemistry people used to consider it suicidal. And it's a place where vanity metrics galore. There were masters who actually conducted tests and enrolled only students who fared very well in that. Oh! the irony. And every time the result came out someone from the tuition centre would come in with a proud face and announce it aloud “35 or some number of centums from our centre this year!”. And without a clue, we would all be awestruck and proud (I still don’t know for what reason) and drooling about that number. This is all without having any clue about — how many appeared for the exam from the tuition centre? What percentage of them got centum? How was the overall state result? What is the share of the percentage of people who scored centum? What is the percentage of centum among the people who did not attend tuition? Was the question paper easy that year? What is the percentage contribution of the schools from which these students are from? Has the centre actually performed better this year compared to last year? What is the yearly trend like? What’s the average score like? Whether the tuition centre really made an difference? Without anything of these striking us, that single number was enough to captivate our minds as well as our parents and convince that we were in the best place. What a vanity!

Somehow I got done with my schooling and tuitions and cleared my board exams. I had scored decently but while comparing to others my marks looked poor. I frankly did not have a metric to choose a college because the metric I had (Marks) itself wasn’t great. Finally, I settled for a college which settle for my marks. But people who had good marks or money had a metric for choosing a college, which was whether that college had campus placements. Again a binary metric. It defined whether large companies came to college and picked candidates in large numbers. When I was persuing my degree this binary metric underwent a lot of transformation. Whether campus placements were available or not became whether all the students from this college got placed? What’s the highest package offered for a student from this college? What’s the mean/median salary? The number of students with multiple offers. In spite of all the transformation, it remained a vanity. One funny thing is none of the colleges published or advertised all these metrics. They cherry-picked only those metrics that showed them in a good light. Many college managements also made sure that they did not publish or let a word out about metrics such as totally how many students are appearing for interviews, whether the percentage number of students who got placed was calculated based upon the total number of students or the number of students who were eligible for placement and what’s the minimum salary that is considered as a threshold for a job etc. In most of the cases, the guy with Rs 8,000 per month salary from a college which charged more than a lakh rupees as fees per year and the guy who got an offer with a decent salary, were considered equally placed as it was binary metric. First of all measuring goodness of education based upon the ability to land a job itself is somewhat vanity. That along with these metrics make it vanity squared.

After college, I entered the corporate world of big glass buildings. I did not know much about the business model of the company or how good the company is performing. For me, a metric to decide whether a company is good or not is purely by the size of the company and the size of its buildings. Bigger means better. I was least concerned about the nature of the job. Given that metric, I did join a very good company. But I did see the amount of time spent in the company still being a proxy for the amount of work one did. Amount of time spent could be a good proxy measure for a worker in the production line where the amount of time you spend is directly proportional to the number of products you produce. But using the same measure for quantifying knowledge-based work is very poor. Some do stay up late and get their work done and some do leave early but on the way back they keep thinking about the problem, sleep over it and fix it the first thing the next day. Using time as a metric will give very contrasting results for these two people as you can’t measure time spent by a guy lingering about a problem and you can only measure how many times you have swiped in or out. When we can’t measure something we resort to some metric that is measurable and think of that as a good measure but it is not. Some time into my job I learned that I was using a wrong metric all along and the size of the company had no relation with the goodness of the job.

After a few years of work, I moved to Bangalore, known as the Silicon Valley of India. Thank god unlike my hometown it's not known as the Silicon Valley of South-East northern part of central India. Staying truest to its name there are a lot of tech companies here. In fact, many areas and landmarks have techie names like the electronic city, sony world signal, ISRO Layout, Satellite town etc. I joined a start-up and I got the taste of some very good data work. During my initial days in Bangalore, I used to roam a lot inspite of the traffic. I did come across a lot of people from varying work backgrounds in coffee shops and some fancy restaurants with a bunch of stickers of elephants, snakes and whatnot on their laptops. Each thing represented a programming language or some conference they attended. I used to have the idea that attending conferences is a big thing and its a thing for nerds and geeks. And for people whom I don’t know personally, I used to consider the number of stickers they had on their laptops as a proxy measure of how knowledgable or intelligent they were. Only after attending a few conferences I understood that they keep a bunch of stickers even before you reach the registration desk and you need not even attend conferences to get those on your laptops. Plus there are a handful of websites to purchase stickers of whatever programming animal you want and put it on your laptop. What a vanity!

After spending a few years in Bangalore and working with data, I have come to somewhat peace with Vanity. Whenever I am bulldozed with some stats, metrics or numbers I always focus on what is missed, because that's where most of the truth lies. The feeling of awe that I used to get by looking and assuming things just by their face value is slowly fading away. These days when I come across posts in LinkedIn with photos of neatly arranged desks with 15 goodies, 2 hoodies and a coffee mug with some company logo on it claiming to be an awesome place to work or videos of employees shaking legs to a tune on a Friday evening claiming to be a fun place to work, I can’t help but smile.