The Modern Alchemist: Turning Data into Gold

At the turn of the millennium, powers lying dormant deep within the realms of mathematics started to stir. The incredible effect of Moore’s law started to surface, with processing power increasing exponentially in the integrated circuits. This power awakened the concepts in statistics and mathematics that were considered too computationally intensive at the time. Now we know these concepts as machine learning, data science and AI. These concepts are not new, they have been around for a very long time. The hot term in AI today, known as the Artificial Neural Network was actually conceptualized in 1958 and was known as the Perceptron algorithm at the time. Mathematicians have been working hard to formulate algorithms to understand and manipulate data for a very long time, and with immense computational power available today we are finally able to appreciate the beauty of decades of hard work. Research and development in the field of AI have really picked up with a tremendous amount of funding from governments and corporations alike. The race to perfecting the AI bears a kind of semblance to the space race in the 70s and 80s. It is this spirit of competition that encourages great strides in technological development, it is a driving force unlike any other.

Many people have called “Data Scientist” as the sexiest job in the 21st century. Well, there are a lot of factors why that perception is valid. Every company is trying to market their products as having some sort of AI, even the most rudimentary products boast having AI in one form or another. 95 percent of the time — this is absolute rubbish. Majority of the people claiming to use AI in their products don’t know what they are talking about, they are just jumping on the marketing bandwagon. This inflation of reputation really hurts the careers of actual data scientists who have poured years in perfecting their art. This also results in an unnecessarily long selection process, since it is vital to separate job seekers who claim to be data scientists and who actually are. So, you may here there are more job openings in the field of data science than ever, but be aware that you will need to go through an arduous selection process to get a decent job as a data scientist.

The position of Data Scientist comes with lots of perks though, with liberal amounts of vacation days, amazing healthcare benefits, and generous compensation and bonus packages to boot. Looking at the way the future is shaping up, it would be safe to say that people in the field of Data Science are going to be considerably well off as compared to their counterparts, in terms of job prospects and financial security. Having ascertained the proposition of a dazzling future in the field of Data Science, let us take a look at exactly how these data scientists use alchemy to turn data into gold.

Let me tell you a story. There was a guy called Jonathan Goldman, who joined a business networking startup called LinkedIn in June of 2006. At the time, LinkedIn had 8 million active accounts. All development teams were dedicated to scaling up the infrastructure to accommodate flawless functioning of the LinkedIn ecosystem with the kind of surge they were experiencing in the number of active users. But what they were all missing in all this excitement, which was very obvious to someone with a little bit of data analysis know-how and a whole lot of common sense, was the behaviour of the users at the time. The users who logged on to LinkedIn were acting like that odd member of the group sipping cocktails at the far end of the table. They were coming to the site and were facing difficulties in forming connections since they had to bring their own connection lists and import the data to Linkedin at the time. This is how the connections were initially formed.

Now, Jonathan Goldman had a PhD in Physics, and like all physics peeps out there he was well versed in the art of data analysis and he had some common sense to boot. Putting the pieces of the puzzle together, he started experiments on recommending connections based on the triangle closing approach. What the triangle closing approach means is that if A and B know C, then it is very likely that A knows B. It is a very simple method. The results started showing amazing conversions in the network formation with networks expanding at an unprecedented rate. This was one of the things that made LinkedIn a wildly successful business network, as you can see today.

This is all what Data Science essentially is. You take a problem — understand it thoroughly — make inferences about the behaviour of the variables in the system and try to solve it with a bunch of tools and methods. Since we have Google now, you don’t even need to know how each of the methods work in excruciating detail. Just know when to apply a method, for what purpose and what are the limitations of the method. What you need the most, is common sense (It is a surprisingly uncommon thing in people these days). This sense or intuition for solving data science problems is developed and sharpened with practice and with solving tons of diverse problems existing all around us.

So, if you want to be a Data Scientist, don’t be intimidated by jargon like data wrangling, data mining and what not — it’s not that complicated. Take up a bunch of online courses from Coursera, edX or Udemy and get coding! It’s really fun when you learn how to play with all the tools :)

…

Job Memos is an initiative taken by the team at Jobmates, where we are working hard to make lightning fast connections between job seekers and recruiters. With one common application you can reach to thousands of recruiters in one click! go to Jobmates.com to learn more

Originally published at www.jobmemos.com.