Hack Your Career: How to Use Data to Monitor Job Markets

Alice Corona
Jan 26, 2016 · 8 min read

I recently gave a tutorial on how monitor job markets by scraping data from AngelList and pushing it into the data publishing platform Silk. Here’s the step-by-step tutorial on how to analyse the growth hacker job market to see who’s hiring and to discover employment hubs, most requested skills, average salary and most common job benefits.

(Of course, this isn’t just useful for growth hacker jobs. You could insert any other job description and follow the same instructions to create your own Silk research tool for jobs for data scientists, Java developers, HR specialists, or anything else.)
Note: Silk has been discontinued as of Dec. 15th 2017 so links are broken and visualizations are static. Will replace them asap.

Image for post
Image for post

Here’s our full end result, rich with insights and interactive data visualizations to explore the growth hacker job market according to AngelList. Use it as inspiration and replicate with other jobs!



1. Create Your Dataset

a) Get a List of AngelList Job Offerings for Growth Hackers

(Note: If you have an AngelList API Key, APIs are the best way to create the dataset!)

Image for post
Image for post
AngelList job search query for “growth hacker”
Image for post
Image for post

3. Copy the results to the clipboard, then create a Google Sheet and copy these job offer URLs in the first column.

4. Repeat Step 2. and 3, copying the results in the columns next to the first one, with the following XPaths as well:

Image for post
Image for post
Resulting Spreadsheet

5. Create another sheet in your Google Sheet and use it to get a list of all the companies hiring and details about them. To fill in this sheet, repeat the step 2. and 3. with the following XPaths:


2. Clean Up Your Dataset

a) Clean up your “Jobs” Dataset in OpenRefine

3. Clean the “Tags” Column

4. Clean “Remote?” Column

5. Add a column with a link to the hiring company for each job offer: Column “Job Offer Link” -> Add Column Based on This Column -> Name Column “Hiring Company Link”; Expression: value.split(‘/jobs/’)[0]

6. Export your project as a CSV and import it into your Google Sheet as the “Import Jobs” sheet.

b) Clean up your “Companies” Dataset in OpenRefine

6. Export your project as a CSV and import it into your Google Sheet as the “Import Companies” sheet.

c) Link the Two Sheets by a Common Column

Image for post
Image for post

3. Create a unique ID column by combining the Job Title with the Hiring Company

Image for post
Image for post

d) Currency Conversion in the “Import Jobs” Sheet

Image for post
Image for post

4. Go back to the “Import Jobs” sheet. Convert all min and max job compensations to dollars, using the VLOOKUP formula to find the exchange rates. Formulas used in this case:

5. Set the format of the cells in these columns to Currency: US $

6. Recalculate a “Job Compensation in $” column by joining the min and max compensations with a “ — “.

7. Calculate a “Median Compensation in $” column with the MEDIAN function

Image for post
Image for post
Result

3. Analyze and Visualize Your Dataset

a) Import Your Dataset in Silk

b) Start Looking for Patterns and Interesting Visuals


4. Publish Your Data-Driven Story

What you can do:

2. Automatically turn each datapoint into a story using “pinning “and templates

3. Embed visualizations on other platforms (Medium, Tumblr, your own blog..) or share them on social media (Silk supports Twitter cards!)

4. Invite other people to collaborate on your Silk


Here’s our end result! Use it as inspiration and replicate with another job market!

SILK STORIES

Silk is a place to explore the world through data.

Alice Corona

Written by

Stories with data, from the data collection (or scrape) to the data visualization. Data storytelling instructor. Currently project leader at batjo.eu

SILK STORIES

Silk is a place to explore the world through data. Each Silk contains data on a specific topic. Silk displays data as beautiful interactive charts, maps and web pages.

Alice Corona

Written by

Stories with data, from the data collection (or scrape) to the data visualization. Data storytelling instructor. Currently project leader at batjo.eu

SILK STORIES

Silk is a place to explore the world through data. Each Silk contains data on a specific topic. Silk displays data as beautiful interactive charts, maps and web pages.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store