Searching Candidates’ Feelings: On the Value of Community Events at Kiwi.com

13 min readJan 13, 2020

This analysis works with two datasets exported from the customer relationship management (CRM) system Beamery and the applicant tracking system (ATS) Jobvite. Both datasets contain historical data for the period of 2017–2019. This analysis aims to examine the candidate flow in the hiring process at Kiwi.com.

Data and why to work with them?

This analysis builds on two articles written by Lucie Chladová. Lucie first analyzed the recruitment process at Kiwi.com; later, she also outlined the issue of measuring recruitment campaigns. I have worked with Lucie on the Beamery implementation that took place in April 2019. With the CRM running, we had the data we needed to finally confirm or disprove the success of community events in the recruitment process focused on developers. The article is based on a conjoint project that I was working on together with Denisa Košutová. Denisa is the creator of the Power BI dashboards presented later on in the article.

Where did the data come from?

Kiwi.com Community team was founded in 2017. Since then, the team has organized a large number of events, including meetups, conferences, educational weekends, bootcamps, and workshops; they’ve also sponsored numerous conferences. The actual number of events for 2018 can be found here. At each of the events, the Community team acquired a number of contacts to people who were interested in talking to the recruiter. These people either wanted to start working at Kiwi.com right away or just learn about job opportunities and job openings. However, all this was happening at the time when Kiwi.com had no CRM system available and so the good old Google Sheets were used instead. Three years after the Community team started organizing events, the Beamery tool was introduced at Kiwi.com. It is sufficient to say that people at Kiwi.com swiftly nicknamed Beamery “Google Sheets on steroids.” Beamery was expected to help with recruitment and community management across the company, becoming the main tool for the Recruitment and Community teams. During the implementation, all historical data were transferred to Beamery; at that time, we have learned that a large portion of data was missing. There are currently around 90 tagged events in Beamery. However, in 2018 alone Kiwi.com organized over 160 events.

Dictionary

For a better orientation, here’s a list of terms used in the article.

Events: all events organized by Kiwi.com or events in which Kiwi.com participated
Beamery: CRM system containing data about people who attended a Kiwi.com event and wanted to be contacted by HR. Beamery also contains contacts sourced by recruiters, but those are not included in this analysis
Jobvite: ATS system containing data about people who have entered the recruitment process
Source type: the source origin of the candidate in Jobvite
Contact: a person who visited an event and wants to be contacted by HR (without a specific time frame in mind)
Lead: (status) a person who visited an event and wants to be contacted by HR immediately
Passive: (status) a person who visited an event and wants to be contacted by HR within a year
Candidate: a person who entered the recruitment process by applying for an open position

What’s in it for you?

In this project, we have examined the HR flow of candidates sourced through community events organized by Kiwi.com. In our context, the term community event candidate stands for developers or employees working at the Engineering department at Kiwi.com. Our main goal was to find answers to the following two questions. First, how many people were so charmed by the community events and company culture that they gave their consent to be contacted by the recruiter (1). Second, whether these people were tagged with the correct source type “Events” in Jobvite (2). Why bother looking into such data? Let’s say that our premise states that events are good because they bring us contacts for future employees. Then, naturally, there has to be a way of proving this through data. In other words, it has to show somewhere. In our case, each employee contact acquired successfully through one of the community events should be tagged in the Jobvite column called Source type with the keyword “Events.” But, let’s face it, we are not living in a perfect world and the reality is much more complicated. Cases in which a candidate was wrongly tagged appear to be quite common. Imagine for example a situation in which someone visits a community event. They join the event, have fun, and later that evening decide to go on the career site jobs.kiwi.com and apply for an open position of their own volition. In such a scenario, neither the Community team, nor the recruiter have the slightest chance to initiate contact with the candidate first. This situation then leads to a wrongly assigned source type in Jobvite; a candidate who applies on their own after attending a community event is marked with a keyword “Career Site” instead of “Events”.

Later, when the company decides to go through the collected data, the “Events” metric has already lost its corresponding value. Not knowing this, the company could wrongly assume that events do not bring them many candidates, while the career site is a successful source of candidates. Based on this finding, the company could decide to cut down the budget for events and pump money towards the career site. When checking the statistics again after a year, the company would most likely discover that the career site is actually not such a great source of candidates and that they’ve lost their long-built developer community because of the lack of events.

KPIs — because everyone loves KPIs

Finding the direct and indirect candidate conversion (%)
Direct conversion: percentage of candidates who are tagged with the correct source type in Jobvite (“Events”)
Indirect conversion: percentage of candidates who are tagged with an incorrect source type in Jobvite and we know for sure that they actually came from events.
Action item for the company: correcting the source type in Jobvite.
Why? More accurate tracking can help us determine how successful events are in the hiring process.
Determining the quality of the candidates (%)
This metric was consulted with the Recruitment team and the criteria were set as the number of candidates who successfully pass the Phone Screening. In other words, this metric means that the candidate successfully moved forward in the hiring process after a scheduled call with the recruiter.
Why? If we know that candidates from a particular source are relevant, we can push this type of hiring. This also leads to a better use of recruiters’ time. Pushing the right type of hiring will result in easing up recruiters’ workload; instead of searching through an endless pile of non-relevant CVs, recruiters can focus on spending time with relevant candidates.
Find the average number of events that the candidates visit before they apply for a position (%)
Why? If we know what precedes the actual job application, we can better schedule the events, cross-promote the events, and manage the candidate flow.

What else did we want to learn?

Percentage of candidates from the source type “Events” in the whole engineering department.
The ratio of lead vs. passive candidates
Candidate’s journey — how long does it take to go through the hiring process stages.

Our Beginnings, or The Tools and the Struggles

It’s going to be easy; it’s just two tables with no data cleaning needed.
– Me before the project
You don’t need a data model for this project. Sometimes it’s useless.
– Anonymous during the Hackathon

As you can imagine, it turned out that our datasets were not as clean as we originally thought they were, and that we indeed need to create a data model when visualizing in Power BI. But I won’t bother you with all the hardships we encountered when conducting the analysis; I will just point out the ones that surprised me as someone who was using Python and Power BI for the first time in life. You are about to read a subchapter called “The Beginner’s Data Analysis.” Feel free to skip this part to the “Back to Business” chapter where the analysis was already done or even further to the “Results” part where you can read more on our findings.

The intermezzo of an emerging data analyst, or how we got stuck and unstuck again

CSV may not always be only comma-separated value, sometimes it can be a semicolon as well. And sometimes it is not a separated value at all! Such were the imports from Beamery. Excel put all the data into one column, and for quite some time we were looking for a solution to this problem. It turned out to be a simple one — uploading everything into Google Sheets did the trick. One of the things we had to decide at the beginning of the project was what languages and tools we’re going to use. In the end, I chose Python and my colleague chose Power BI. Python became my choice number one because I wanted to learn it and also because it is Mac-friendly. The great advantage of working with different tools is that we could compare our results to avoid any mistakes.

Back to business

First, we needed to create a nice and functional data model. We have sneaked into one of the meeting rooms with the biggest whiteboard we could find; whiteboards proved to be our best friends when it came to data modeling. Once the model was “working” on the whiteboard my colleague had taken over, transforming it into a Power BI data model with no many-to-many (M:M) relationships. One thing we learned at Hackathon: many-to-many is definitely not a good thing.

Data model on large whiteboards is the best

In the meantime, I poured myself into Python, opening the CSV files. Interested in whether I used Pandas? That’s a funny question. The answer is: no, I did not, and for two reasons. Firstly, since I have already figured out how to open everything without Pandas, it simply didn’t make sense to use Pandas. I would have to spend extra hours on the project just to do it right, so to speak. Secondly, I wanted to use the coding language without any libraries to practice the language and understand how it works without any external help. This being said, I’m not bailing on Pandas. I definitely want to learn how to use it in my next project. Let’s now use Hermione’s time-turner and move on to the results of the project. Oh, and if you are interested in my script, feel free to have a look at my GitHub.

import csv
from csv import reader

Results

TL;DL The research confirmed the belief of the Community team: events are a successful hiring tool. At the same time, it confirmed Lucie Chladova’s thesis which said that the candidate source tag is often wrongly used.

KPI 1: Direct and indirect conversion

Direct conversion: 70.94 %
Indirect conversion: 29.06 %

The numbers above prove that out of the 100% of the candidates coming from events, 30% have an incorrect source type assigned in Jobvite. Keep in mind that these numbers are just a fraction of the actual total number. At the beginning I wrote that:

There are currently around 90 events tagged as a source in Beamery. However, in 2018 alone, there were over 160 events organized.

We have to take into account the disproportion caused by the lack of data. In fact, the number of events throughout the past three years would probably fivefold, meaning that in Beamery there should be a record of at least 450 events. The remaining question is how much this would influence the direct and indirect conversion and what would be the final count in absolute numbers. Now that Beamery is being used, all candidates are tracked. HR has more accurate data about where the candidates are coming from, what positions they are applying to, who was the last person in touch with them, and how successful the candidates are in the hiring process.

KPI 2: Candidate Quality (%)

The fact that events are bringing candidates to Kiwi.com does not necessarily mean that these candidates are the quality ones. If a recruiter receives 1000 CVs in a month and only 10 are relevant candidates for the position then the success rate is only 1%. In Kiwi.com the hiring process, in a nutshell, looks like this: HR gets a CV, the recruiter goes through the CV, “screening” the candidate and if the candidate fits the position then the recruiter schedules a phone screening with them. The phone screening helps the recruiter and the candidate make sure that they are on the same page. During the phone screening, both can ask follow-up questions to clarify the job description, and decide whether they want to continue in the hiring process. If everything goes well and the candidate successfully completes the phone screening then they are given the label quality candidate. Every time a candidate enters in a new stage in Jobvite, the ATS automatically marks the date and time. Because we need to learn up to what stage the candidate usually gets, I have created a dictionary in Python that assigns the highest achieved stage for each candidate in Jobvite.

success_dict = {}for row in events:
    mail = row[4]
    source = row[5]
    new = 0 if row[7] is "" else 1
    screened = 0 if row[8] is "" else 1
    phone_screen = 0 if row[9] is "" else 1
    practical_task = 0 if row[10] is "" else 1
    reference_check = 0 if row[11] is "" else 1
    hiring_manager = 0 if row[12] is "" else 1
    first_interview = 0 if row[13] is "" else 1
    second_interview = 0 if row[14] is "" else 1
    offer_sent = 0 if row[15] is "" else 1
    hired = 0 if row[16] is "" else 1stages = [hired, offer_sent, second_interview, first_interview, hiring_manager, reference_check, practical_task, phone_screen, screened, new]
    for i in range(len(stages)):
        if stages[i] == 1:
            success_dict[mail] = len(stages) - i
            break

What remains to do now is to calculate the percentage of candidates who achieved stage equal to or bigger than 4. Entering stage 3 does not necessarily mean that the candidate successfully went through the phone screening, hence the stage 4 condition.

def keys_by_values(some_dictionary, list_of_values):
    list_of_keys = list()
    list_of_items = some_dictionary.items()
    for item  in list_of_items:
        if item[1] in list_of_values:
            list_of_keys.append(item[0])
    return  list_of_keyslist_of_keys = keys_by_values(success_dict, [4,5,6,7,8,9,10,] )
print("Percentage of quality candidates:", round(( len(list_of_keys) / len(success_dict) * 100 )), "%" )Percentage of quality candidates: 48 %

Out of the total number of community candidates, 48% are marked as quality ones.

KPI 3: The average number of events candidates visit before applying for a position

Kiwi.com Community team believed that there is indeed an event funnel which precedes the hiring funnel. They had a theory based on observations according to which some events bring more job applicants than others. They thought that before a candidate attends one of those candidates-bringing events, they visit some other, more general event like a meetup or a conference. After the first initial contact at a conference or a meetup, the candidate would then attend an education weekend or a Summer Camp.

Mean: 1.0
Max value: 4

However, the data shows that the average candidate attends only one event before submitting a job application.

Other findings

Lead vs passive candidates

Another interesting number for the Recruitment team is the total number of contacts in Beamery with the status lead or passive. There are several statuses in Beamery that can be used but the Community team uses primarily those two. Status lead informs the recruiter that the person would like to be contacted right away. Status passive tells that the person wants to be contacted in the future (usually within one year). The total number of both statuses tells the recruitment team how many potential candidates there are.

Number of leads: 283
Number of passives: 446
Number of other statuses: 1241

In total there are 729 contacts from the events in Beamery, that have not been contacted yet.

Power BI dashboard for HR and the candidate flow

As I have mentioned above, my colleague and I approached the data differently. I have used mostly Python and Denisa used Power BI. Our decision turned beneficial since it allowed us to compare our findings. Big thank you goes to Denisa, since the dashboards presented here are her work.

The dashboard below enables to filter data based on its source type. After you click on source Events, the dashboard changes and the hiring funnel shows the percentage of successful candidates in different hiring stages. In case of the events, the percentage of Hired is 13,04 which is 8,04% more than from all other sources combined. You can see that the most successful hiring source is the Hiring Manager (42%), followed by Internal (18,31%) and Employee Referral (17,62%). Events ended up in the fourth place. Keep in mind that internal transfers will fill in an open position but at the same time they will usually open a new one.

Candidate flow chart

The flow chart shows with what source type candidates enter the hiring process and how their journey looks like all the way through to the hired stage. For better readability, rejections statuses are omitted (Rejected, Offer Rejected or Candidate Withdrew). The flow chart illustrates how many people entered a particular stage and moved to another one. The thinning of the lines represents the decreasing number of candidates who successfully moved onto a new stage. Notice the large number of candidates from the source Job Boards who did not advance to the phone screen and ended up in the Non-Phone Screen right away.

Conclusion

This analysis confirmed that community events in Kiwi.com are an important hiring tool. Even though events are not bringing as high volumes of candidates into the company as job boards, events bring quality candidates. Since CRM Beamery was introduced as a new tool at Kiwi.com, the tagging and managing of contacts became easier. As a result, the source type is now properly tagged. Thanks to that, the indirect conversion is smaller and the Recruitment team knows from what source the candidates are coming. On the other hand, it is important to remember that community events should not only be seen as a hiring tool. Community work is important for PR, employer branding, networking, etc. As an example, consider the Global Travel Hackathon or the Women in Tech conference. Developers visiting those events will most likely post on their social media, talking about the events within their community. Thus, the circle is closing.