LinkedIn Job Trends
Mentor: Sining Chen
LinkedIn has a community of professionals and is also a fitting online platform for job hunting, especially during a pandemic. In this story, we saved a data set of the number of job postings for the 35 job functions that LinkedIn is able to filter. These “functions” include broad categorizations for jobs like “design”, “business development”, “engineering”, and more. By visualizing this data on line graphs and pie charts, we could observe similarities between the types of jobs and also gain insight into the distribution of the categories or jobs.
The technical specifics of our process can be found at the link to our other story where we explain how to use Selenium and Beautiful Soup for scrapping data or performing automated functions: https://medium.com/@sophie14159/linkedin-scrapper-a3e6790099b5
Our Dataset
As a Canadian resident, the information I gathered with our program focuses on jobs located in Canada. With this location filter on LinkedIn, I ran my code to search these following types of jobs on LinkedIn.
Below is the list of how many job postings are categorized in a specific type of job function that LinkedIn filters. This is an example of the data gathered from August 15
2020-08-15 Management 53147
2020-08-15 Manufacturing 49236
2020-08-15 Information Technology 44666
2020-08-15 Other 36069
2020-08-15 Sales 34510
2020-08-15 Business Development 24167
2020-08-15 Health Care Provider 19360
2020-08-15 Engineering 18664
2020-08-15 Customer Service 11247
2020-08-15 Finance 10042
2020-08-15 Administrative 8847
2020-08-15 Marketing 6185
2020-08-15 Design 5374
2020-08-15 Education 5210
2020-08-15 Training 5124
2020-08-15 Art/Creative 4919
2020-08-15 Project Management 4669
2020-08-15 Accounting/Auditing 4510
2020-08-15 Analyst 4108
2020-08-15 Science 3354
2020-08-15 Human Resources 2633
2020-08-15 Quality Assurance 2485
2020-08-15 Consulting 1983
2020-08-15 Research 1915
2020-08-15 Writing/Editing 1814
2020-08-15 Public Relations 1738
2020-08-15 Legal 1671
2020-08-15 General Business 1391
2020-08-15 Supply Chain 1173
2020-08-15 Product Management 897
2020-08-15 Purchasing 728
2020-08-15 Production 698
2020-08-15 Strategy/Planning 684
2020-08-15 Distribution 303
2020-08-15 Advertising 174The data is stored in a text file that can be accessed in order to visualize the data onto either a series of line graphs or combined into a single one.
Visualizing the Data
With data gathered daily from August 15th to September 11th, we obtained graphs that demonstrated the trends of each job function. For example, the graph for “Health Care Provider” is posted below.
Despite using a smaller dataset, we could make observations from the 35 categories.
1. Similar trends
Although there are more jobs related to manufacturing function as opposed to management, it was interesting to see that their graphs almost seem to be a copy and paste of each other as shown below.
Something similar can be said for the job postings in the finance v.s. sales sectors. This is interesting because one might associate finance and sales to be in a similar grouping of jobs, except their graphs don’t match as exactly as manufacturing and management do.
What can most obviously be observed is the common trend of subtle fluctuations that somehow led to a sudden rise in the number of postings from September 1st to 2nd. It is unfortunate that our data does not allow us to explain such changes in the trends!
2. General Consistencies
With all the different job functions, we decided to plot them on the same graph in order to compare them more easily. We used a logarithmic y-axis scale so that the trends we visualized would be of the same scale.
The graph above looks busy due to the large number of lines we plotted. However, it is easy to observe that, spare the few abnormal fluctuations, the general trend of the number of job functions appear to remain flat and consistent.
Gathering Data — Courtesy of Selenium and BeautifulSoup
Going back to the data gathering process, our major tools were Selenium and BeautifulSoup.
Selenium is a web-based automation tool that works well with Python. In our program, Selenium allows us to access a browser (Chrome) and web pages, specifically to reach LinkedIn pages.
We combined this tool with Beautiful Soup, which is a Python library that can scrape texts from the source code of a web page. The source code can be accessed with a simple right-click to find the Inspect (CTRL+SHIFT+I) option as shown in the figure on the left. Accessing this source code is the key to using Python commands to navigate in a web page and to scrape the data we needed.
More about the technical specifics of our process, you may find segments of our code and other explanations at the link to our other story which is linked near the top of the story.
Summary
The information we could gain from a web page using Python code enabled us to visualize job posting trends on LinkedIn from the number of postings per job function over time.
