Our first steps in AI
In Jit Team, we have an internal system called WeJit (about which I wrote before here and here). It stores data about our clients, projects, company events and most importantly — people. Everyone has a profile that is just like a resume — it contains information about where we studied, worked, what skills we have, etc. Having around 2 thousand people (Jit employees and screened candidates from the job market) in the database gives us lots of possibilities when it comes to providing competent people for our clients.
But at some point it starts to get hard to rely on people just knowing who to choose for each client. Of course WeJit has a fulltext search engine, lots of filters to search by, etc., but what if one of your developers quit and you need to find a replacement as quickly as possible? Or if one of your clients needs a small and quick project and time is a big factor? At some point there is just too much data to memorize or browse.
On the other hand, all this data can be a big advantage. We decided we want to use it and build a profile compatibility mechanism. A service that would take in a profile or job description and suggest a number of similar profiles that would be a good match. It was a great opportunity to create something useful and try to fiddle with something new. And hence, our first steps in the field of AI.
Our general plan was
1.All the relevant fields from each section of the profile are joined into one string.
The profile as shown on the screenshot above is divided into multiple sections. Not all information stored in the profile is relevant to the person’s skillset. The names of clients and projects are examples of information that don’t give us information about whether someone is a frontend, backend, java, C# or Python specialist.
2. The result string is processed by removing stop words and performing lemmatization.
Even when we take only the fields that contain the most relevant data we still get words that don’t add any value to the evaluation of someone’s skillset. That’s why we use techniques that filter out the most important words and migrate them to their root words.
For example let’s take the sentence
“As a Java developer I worked for a financial institution using Spring and Postgres.”.
After cutting out stop words the sentence might look like this (depending on the set of stop words we would use):
“Java developer worked financial institution (bank) Spring Postgres”.
Lemmatizing means transforming as many words as we can to their base form so it might look like this:
“Java developer work finance institution Spring Postgres”
thus taking out the essence of the sentence.
3. Each word that is left is converted to a vector using a word embedding model.
A word embedding model can basically be a large set of words with their vector equivalents given some scale. Let’s try to simplify this and boil it down to our example and say that we are looking for a Java developer that is experienced in the finance industry.
We can look at the words we have and give them 2 ratings:
- is it a java skill?
- is it a finance related skill?
Our scale will be 0–1:
I did some assumptions here, for example Spring is a very popular Java web framework, but theoretically using the word “Spring” in you resume doesn’t HAVE TO mean the framework. But it very likely does. So that’s why I gave it a 0.9 rating in the isJava parameter. Also — if it is a very popular Java framework and Java is often used in FinTech I added 0.5 to the isFinance parameter.
Now, we can take only the words that have relevance to our scales:
If we interpret each pair of numbers as a 2 element vector, we have a model that lets us judge if someone could be suited for a Java developer job in finance.
4. A mean vector is calculated and saved for the profile.
That’s actually quite simple math:
(1+0.5+0.5+0.9+0.65)/5 = 0.71
(0.5+0.8+0.9+0.5+0.5)/5=0.64
So the person that wrote this sentence is quite suited for the position we are hiring for — his vector is [0.71, 0.64].
One thing to keep in mind here is that even though the scale of our model is 0–1, the results probably won’t be exactly aligned to that scale. The best suited people might have their vectors somewhere around the 0.6–0.7 spectrum so you have to adjust to that if you would like to present some kind of score.
5. Matching. When looking for people that are most suited for a job description we look for vectors that are closest to the vector derived from the description.
Assuming we have 3 people in our database:
A [0.67, 0.30], B [0.24, 0.58], C [0.33, 0.17]
we can visualize these vectors as points on a 2D plain:
Our clients contacts us and tells us one of their developers is leaving them and they need a replacement. We get their resume and after processing it we get the score [0.59, 0.47]. Looking back at the plain, we can easily see that our employee A is the most suited for replacing this person:
The next step is applying this math to a database. It would be hard to put 2000 points all on a map and eyeball the distances between the points so we need to use the formula for calculating the distance between two points.
Here are the results:
We can clearly see that it is very easy to rank the people most matching to our original developer by the distance calculated with our formula and therefore we have a valid comparison mechanism for our database.
Keep in mind this was our first approach—something simple to have fun with and create something that would be useful for our HR department. Having some more experience now, we are evolving this solution and creating something new that will be more accurate which will probably be the subject of one of the next articles published by Jit Team members. :-)
