An LLM agent that builds and maintains a job title taxonomy
We’ll set up an LLM agent that can be fed an unlimited amount of job postings, which it uses to build a taxonomy of all job titles. The taxonomy will include job titles, synonyms of each job title (plural forms, abbreviations, variations in spelling) and a summary of the tasks, responsibilities, qualifications and work area commonly associated with the job title.
Why an LLM agent is a good solution
This section explains the main concepts, the problem at hand and why an LLM agent is a specifically smart way to solve this problem. If you just want to get to the code, feel free to skip ahead.
Job titles and job title taxonomies
A job title is often the best first order approximation of the content of a job. A job title taxonomy is a table of job titles and might include a main job title, commons synonyms and a description. By adding more relations between job titles, e.g. to what degree their labor markets overlap, you can make them more advanced. Job title taxonomies are used by job boards, labor market authorities and analysts working with the labor market. There are several ways to construct a job title taxonomy, in this article we’ll build one using online job postings.
Why is it difficult to automate the building and maintenance of a job title taxonomy?
Building a job title taxonomy from online job postings normally requires skilled employees who read through job postings and evaluate the existing taxonomy. As the taxonomy aught to contain the thousands upon thousands of job titles used in the labor market, it is difficult for each employee to fully comprehend the taxonomy. Splitting the taxonomy up based on e.g. industries risk causing inconsistency or redundancy. And as millions of new online job postings are produced each year, reading through them all and comparing job titles in postings to job titles in the taxonomy is extremely time consuming.
But it is incredibly difficult to automate this task. It is dependent on an employee using common sense and switching between different strategies to determine if a job posting contains a job title that requires changes to be made to the job title taxonomy. The employee’s workflow will look something like this:
- Read through the job posting and identify the job title and the content of the job.
- Check if the verbatim job title from the job posting is in the job title taxonomy.
- Check if the job title from the job posting is a slightly different way to spell a job title already in the taxonomy. Or maybe it is an abbreviation of something already in the taxonomy.
- What if the job title from the taxonomy is a quite different formulation of something already in the taxonomy? E.g. a job posting looking for a “Digital Engagement Coordinator”, which has a description that is identical to a “SoMe-manager”? Identifying this requires reading the content of the job in the job posting and comparing it to the descriptions of each job title in the taxonomy.
- When all of this has been looked up in the taxonomy, it is time to decide if the taxonomy has to be updated:
• Is this a new job title that aught to be added to the taxonomy?
• Is this a synonym of a job title that is already in the taxonomy?
• Is this job title verbatimly in the taxonomy, but the description in the job posting reveals the job title is e.g. being used for very different functions in different industries, implying the job title entry in the taxonomy needs to be split in two to reflect this?
• Is the description of the job title in the taxonomy still accurate? - Finally, whatever is decided in step 5. has to be implemented in the job title taxonomy.
Automating this workflow with “if-then”-code, and achieving a high-quality result, is more or less impossible.
How LLM agents can automate this
LLM agents are large language models that have been given access to “tools”, e.g. the ability to interact with a database. This makes it possible to automate tasks that has hitherto required a human to carry them out.
In this case, the database is the taxonomy of job titles. The LLM agent can take a job posting as an input, identify job titles in it, query the job title taxonomy in different ways, decide if and how the taxonomy should be updated and make the update.
By feeding job postings one at a time to the LLM agent, we can start with an empty job title taxonomy and let the LLM agent discover job titles and add them to the taxonomy for us. As time goes by and new job postings come online, we simply keep feeding them to the LLM agent, which keeps the taxonomy up-to-date.
This task could alternative be split into subtasks and an LLM could solve each subtask independently. This would require a human to specify how to move between subtasks. The advantage of the LLM agents is its ability to decide how to move between subtasks, going back and forth between subtasks as new information is revealed and iterating until a good solution is achieved.
The code
In this section we will set up the LLM agent using Langchain’s framework for agents: https://python.langchain.com/docs/modules/agents/
We will use OpenAI’s GPT-4 Turbo, which you can read about here: https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo
The entire project along with all dependencies can be seen in this GitHub repo: https://github.com/EliasWiegandt/build_taxonomy_with_llm_agent
Set up the LLM agent
First we need to import all the libraries we’re going to need.
import os
import dotenv
import pandas as pd
from fuzzywuzzy import fuzz, process
from langchain.agents import AgentExecutor
from langchain.agents.format_scratchpad import \
format_to_openai_function_messages
from langchain.agents.output_parsers import OpenAIFunctionsAgentOutputParser
from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.tools import StructuredTool, Tool
from langchain.tools.render import format_tool_to_openai_function
assert dotenv.load_dotenv()We then set up our credentials for OpenAI. I use python-dotenv to handle credentials.
openai_api_key = os.getenv("OPENAI_API_KEY")
org_id = os.getenv("OPENAI_ORG_ID")
os.environ["OPENAI_API_KEY"] = openai_api_key
openai_model = "gpt-4-1106-preview"We then load data on online job postings. A search for “job posting dataset” shows that there are many datasets around, e.g. from Kaggle. Alternatively you can just generate some synthetic data using an LLM.
dtype_dict = {
"job_id": "uint32",
"title": "string",
"description": "string",
}
df = pd.read_csv("job_postings.csv", dtype=dtype_dict, index_col="job_id", usecols=list(dtype_dict.keys()), nrows=50)
df["full_text"] = df["title"] + "\n" + df["description"]We then set up our taxonomy. In this toy-example we’ll just use a pandas DataFrame.
index_cols = ["main_job_title"]
other_cols = ["synonyms", "description"]
TAXONOMY = pd.DataFrame(columns=index_cols + other_cols).set_index(index_cols)
main_job_title_definitions = "The main job title used to index the job title entry in the taxonomy"
synonyms_definitions = "Synonyms of the main job title, seperated by commas"
description_definitions = "Very short summary of the job title's most common tasks, responsibilities, qualifications, tools and area of work"Next, we specify the functions that will eventually be the tools the agent can use. We need functions for querying the job title taxonomy and functions for editing the job title taxonomy.
def top_similar_descriptions(df, new_desc, top_n=3):
scores = process.extract(new_desc, df["description"],
scorer=fuzz.token_set_ratio, limit=top_n)
hits = []
for score in scores:
hit = TAXONOMY[TAXONOMY["description"] == score[0]].copy()
hit["score"] = score[1]
hits.append(hit)
return pd.concat(hits)
def look_up_main_job_title_in_taxonomy(main_job_title: str) -> str:
f"""
Check if a job title is registered as a main job title in the taxonomy.
:param main_job_title: {main_job_title_definitions}
:return: pandas DataFrame with stringified slice of taxonomy.
"""
main_job_title = main_job_title.lower()
if TAXONOMY.empty:
return "Taxonomy is empty, i.e. there are no job titles at all in it currently."
try:
return TAXONOMY.loc[[main_job_title], other_cols].to_string()
except:
return "Job title not found in taxonomy"
def search_for_similar_descriptions_in_taxonomy(description: str) -> str:
f"""
Get entries in job title taxonomy that has the most similar descriptions.
:param description: {description_definitions}
:return: pandas DataFrame with stringified slice of taxonomy.
"""
description = description.lower()
if TAXONOMY.empty:
return "Taxonomy is empty, i.e. there are no job titles at all in it currently."
top_matches = top_similar_descriptions(TAXONOMY, description)
return top_matches.to_string()
def add_job_title_to_taxonomy(
main_job_title: str,
synonyms: str,
description: str,
) -> str:
f"""
Add a job title to the taxonomy.
:param main_job_title: {main_job_title_definitions}
:param synonyms: {synonyms_definitions}
:param description: {description_definitions}
:return: String describing whether the job title was succesfully added as a new job title in the taxonomy.
"""
main_job_title = main_job_title.lower()
synonyms = synonyms.lower()
description = description.lower()
try:
TAXONOMY.loc[main_job_title] = [synonyms, description]
return f"Added job title '{main_job_title}' to the job title taxonomy"
except Exception as e:
return str(e)
def replace_synonyms_of_job_title_in_taxonomy(
main_job_title: str,
updated_synonyms: str,
) -> str:
f"""
Replaces the synonyms of a job title with new synonyms.
:param main_job_title: {main_job_title_definitions}
:param updated_synonyms: {synonyms_definitions} that will replace the previous synonyms
"""
main_job_title = main_job_title.lower()
updated_synonyms = updated_synonyms.lower()
try:
TAXONOMY.loc[main_job_title, "synonyms"] = updated_synonyms
return f"Succesfully added synonym(s) '{updated_synonyms}' to '{main_job_title}'"
except Exception as e:
return str(e)
def replace_description_of_job_title_in_taxonomy(
main_job_title: str,
updated_description: str,
) -> str:
f"""
Replaces a description of a job title with a new description.
:param main_job_title: {main_job_title_definitions}
:param updated_description: {description_definitions} that will replace the previous description
:return: String describing whether the operation was a succes
"""
main_job_title = main_job_title.lower()
updated_description = updated_description.lower()
try:
TAXONOMY.loc[main_job_title, "description"] = updated_description
return f"Previous description for '{main_job_title}' replaced by updated description"
except Exception as e:
return str(e)
def delete_job_title_from_taxonomy(
main_job_title: str,
) -> str:
f"""
Deletes a main job title and its associated data from the taxonomy
:param main_job_title: {main_job_title_definitions} that will be deleted from the taxonomy
:return: String describing whether the operation was a succes
"""
main_job_title = main_job_title.lower()
try:
TAXONOMY.drop(main_job_title, inplace=True)
return f"Deleted '{main_job_title}' from taxonomy"
except Exception as e:
return str(e)Next, we use the functions above to create tools for our agent.
look_up_main_job_title_in_taxonomy_tool = Tool.from_function(
func=look_up_main_job_title_in_taxonomy,
name="look_up_main_job_title_in_taxonomy",
description="Lookup a main job title in the taxonomy."
)
search_for_similar_descriptions_in_taxonomy_tool = Tool.from_function(
func=search_for_similar_descriptions_in_taxonomy,
name="search_for_similar_descriptions_in_taxonomy",
description="Find main job titles with the most similar descriptions from the taxonomy."
)
add_job_title_to_taxonomy_tool = StructuredTool.from_function(
func=add_job_title_to_taxonomy,
name="add_job_title_to_taxonomy",
description="Add a main job title, its synonyms and its description to the taxonomy"
)
replace_synonyms_of_job_title_in_taxonomy_tool = StructuredTool.from_function(
func=replace_synonyms_of_job_title_in_taxonomy,
name="replace_synonyms_of_job_title_in_taxonomy",
description="Replace the synonyms of a job title in the taxonomy"
)
replace_description_of_job_title_in_taxonomy_tool = StructuredTool.from_function(
func=replace_description_of_job_title_in_taxonomy,
name="edit_description_of_job_title_in_taxonomy",
description="Replace the description of a job title in the taxonomy"
)
delete_job_title_from_taxonomy_tool = Tool.from_function(
func=delete_job_title_from_taxonomy,
name="delete_job_title_from_taxonomy",
description="Delete a job title from the taxonomy"
)
tools = [
look_up_main_job_title_in_taxonomy_tool,
search_for_similar_descriptions_in_taxonomy_tool,
add_job_title_to_taxonomy_tool,
replace_synonyms_of_job_title_in_taxonomy_tool,
replace_description_of_job_title_in_taxonomy_tool,
delete_job_title_from_taxonomy_tool
]We can then set up our LLM and our LLM agent.
llm = ChatOpenAI(model=openai_model, temperature=0.0, organization=org_id)
llm_with_tools = llm.bind(functions=[format_tool_to_openai_function(t) for t in tools])Most importantly, we need to explain to the LLM agent what we want it to do. I like to also provide it with some examples of what solving a task could look like.
examples = f"""
EXAMPLES
These are examples of work flows and how you use the different tools. Describe you thoughts at each step.
EXAMPLE 1
You are given this (shortned) job posting: 'To pædagoger på 28-30 timer til KKFO'en ved Bispeengsskolen L[...]'
1. You extract the following information:
a. Explicitly stated job title: 'pædagoger' (note, although the job title is plural, if the singular form is not mentioned anywhere in the job posting, you should use the plural form as the main job title in the taxonomy)
b. Synonyms: ''
b. Description: [YOU MAKE THIS BASED ON THE POSTING]
2. You use the tool 'look_up_main_job_title_in_taxonomy' for checking if a job title is in the taxonomy and find that 'pædagoger' is not in the taxonomy.
You then use the tool 'search_for_similar_descriptions_in_taxonomy_tool' for searching for similar descriptions. It returns three results (by default). None are similar to the description of the job posting you made.
3. You decide that 'pædagoger' is a new job title that should be added to the taxonomy. You use the tool 'add_job_title_to_taxonomy_tool' to add 'pædagoger' as a new 'main_job_title', along with the synonyms set to '' and description based on the job posting, as defined above.
EXAMPLE 2
You are given this (shortned) job posting: 'Rodulphi is seeking a Nurse Practitioner [...]'
1. You extract the following information:
a. Explicitly stated job title: 'Nurse Practitioner'
b. Synonyms: ''
b. Description: [YOU MAKE THIS BASED ON THE POSTING]
2. You use the tool 'look_up_main_job_title_in_taxonomy' and find that 'Nurse Practitioner' is not in the taxonomy.
You then use the tool search_for_similar_descriptions_in_taxonomy_tool'. It returns three results (by default). One has the main job title 'nurse', with 'synonyms' set to '' and a 'description' that is similar to the description you made.
3. You decide that 'Nurse Practitioner' is a synonym of 'nurse' and use the tool replace_synonyms_of_job_title_in_taxonomy_tool to replace the current synonyms '' with 'Nurse Practitioner' for the job title with 'main_job_title' equal to 'nurse'.
EXAMPLE 3
You are given this (shortned) job posting: 'Assistant Store Director (ASD) for [...]'
1. You extract the following information:
a. Explicitly stated job title: 'Assistant Store Director'
b. Synonyms: 'ASD'
b. Description: [YOU MAKE THIS BASED ON THE POSTING]
2. You use the tool 'look_up_main_job_title_in_taxonomy' and find that 'Assistant Store Director' is not in the taxonomy.
You then use the tool 'search_for_similar_descriptions_in_taxonomy_tool'. It returns three results (by default). None are similar to the description of the job posting you made.
3. You decide that 'Assistant Store Director' is a new job title that should be added to the taxonomy. You use the tool 'add_job_title_to_taxonomy_tool' to add 'Assistant Store Director' as a 'main_job title' to the taxonomy, along with the synonyms 'ASD' and the description you made.
EXAMPLE 4
You are given this (shortned) job posting: 'Faglig og engageret pædagog til basisteam i[...]'
1. You extract the following information:
a. Explicitly stated job title: 'pædagog'
b. Synonyms: ''
b. Description: [YOU MAKE THIS BASED ON THE POSTING]
2. You use the tool 'look_up_main_job_title_in_taxonomy' and find that 'pædagog' is not in the taxonomy.
You then use the tool 'search_for_similar_descriptions_in_taxonomy_tool'. It returns three results (by default). One of them 'pædagoger' is the plural form of the job title you found in the job posting.
3. You decide that 'pædagog', as the singular form, is a better choice for the main job title in the taxonomy than 'pædagoger'. You use the tool 'delete_job_title_from_taxonomy_tool' to delete delete 'pædagoger'.
You then use the 'add_job_title_to_taxonomy_tool' to add 'pædagog' as a 'main_job_title', along with the synonym 'pædagoger' and new 'description' where you combine the previous description and the new description from the job posting.
EXAMPLE 5
You are given this (shortned) job posting: 'Durkdreven og humoristisk pædagog til botilbud på Mors [...]'
1. You extract the following information:
a. Explicitly stated job title: 'pædagog'
b. Synonyms: ''
b. Description: [YOU MAKE THIS BASED ON THE POSTING]
2. You use the tool 'look_up_main_job_title_in_taxonomy' for checking if a job title is in the taxonomy and find that 'pædagog' is already in the taxonomy.
You check the provided description and find that it is accurate. You then conclude the job title is already in the taxonomy and no further action is needed.
"""And the examples are combined with a description to form the prompt we’ll give the agent:
system_prompt = f"""
You are an assistant that builds and maintains a taxonomy of job titles.
HIGH-LEVEL TASK
You take a job posting and
1. Identify these from the job posting:
1a. Note all explicitly stated (verbatim) job titles from the job posting that the future employee will hold.
1b. Note all explicitly used synonyms of the job title in the job posting (could be plural forms or abbrevations, see further details below)
1c. Make a very short summary of the tasks, responsibilities, qualifications, tools and area of work that job posting describe the future employee will have.
2. Compare the explicit job title(s) you found in 1a.to the job titles in the taxonomy (both verbatim and by comparing descriptions).
3. Choose whether the job title from the job posting represents:
a. A new job title, that you should add to the taxonomy, along with any synonyms you found and the description you made.
b. A synonym of a job title already in the taxonomy, that you should add to the synonyms of this job title in the taxonomy.
c. A job title already in the taxonomy, implying you should check if the description of the job title in the taxonomy is accurate and update it if necessary.
DEFINITIONS
Each row in the taxonomy should be contain a 'main_job_title', its 'synonyms' and a 'description'.
'main job title' is the job title that we use to refer to the entire row in the taxonomy. We prefer non-abbreviated singular forms as main job titles.
'synonyms' is e.g. plural forms of the main job title, e.g. 'nurses' for the main_job_title 'nurse'. Or abbreviations e.g. 'ASD' for 'Assistant Store Director'.
'description' is a very short summary of the tasks, responsibilities, qualifications, tools and area of work that job posting describe the future employee will have.
f{examples}
So again:
- Find the job titles in the job posting
- Check if the job titles are in the job title taxonomy using the tools.
- If they are not in the taxonomy, add them. If they are already in the taxonomy, or something very similar are, decide if the job titles should be added as synonyms or if the descriptions in the taxonomy should be updated.
"""Finally, we add the infrastructure for handling the messages and our LLM agent.
prompt = ChatPromptTemplate.from_messages(
[
(
"system",
f"""
{system_prompt}
Here is a list of all the tools:
{tools}
""",
),
("user", "{input}"),
MessagesPlaceholder(variable_name="agent_scratchpad"),
]
)
agent = (
{
"input": lambda x: x["input"],
"agent_scratchpad": lambda x: format_to_openai_function_messages(
x["intermediate_steps"]
),
}
| prompt
| llm_with_tools
| OpenAIFunctionsAgentOutputParser()
)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, handle_parsing_errors=True)Running the agent
We’re now ready to feed job postings to our LLM agent.
for ix, row in df.iterrows():
agent_executor.invoke({"input": row['full_text']})The LLM agent will start grinding through the job postings in the data set. The output from the very first time the LLM agent reviews a job posting looks like this with my data:
> Entering new AgentExecutor chain...
Invoking: `look_up_main_job_title_in_taxonomy` with `Senior Insights Analyst`
Taxonomy is empty, i.e. there are no job titles at all in it currently.
Invoking: `add_job_title_to_taxonomy` with `{'main_job_title': 'Senior Insights Analyst', 'synonyms': '', 'description': "Develop insights and storytelling, own data analysis and reporting, present insights addressing client needs, apply analytical techniques, create client-ready materials, ensure data quality, design research instruments, analyze research results, write reports, and collaborate with client management. Requires a Bachelor's degree, experience in market analysis or research, proficiency in data analysis software, and strong communication skills."}`
Added job title 'senior insights analyst' to the job title taxonomyThe job title "Senior Insights Analyst" from the job posting has been added to the taxonomy as a new main job title, with no synonyms and a description that includes the development of insights and storytelling, ownership of data analysis and reporting, presentation of insights that address client needs, application of analytical techniques, creation of client-ready materials, ensuring data quality, designing research instruments, analyzing research results, writing reports, and collaborating with client management. The role requires a Bachelor's degree, experience in market analysis or research, proficiency in data analysis software, and strong communication skills.
> Finished chain.The LLM agent is looking at a job posting that advertise a job as a “Senior Insights Analyst”. It looks that job title up in the taxonomy and finds that the taxonomy is empty. It then adds this job title to the taxonomy along with a description of it. In the early stages, the taxonomy is small and the LLM agent spends most of its time just adding new job titles to it.
Later, the agent will start to make more and mor advanced decisions. Here is the LLM agent’s summary of a job posting it worked on, it both edits an existing job title in the taxonomy and adds a new one.
> Entering new AgentExecutor chain...
[...]
1. I checked the taxonomy for the job title "Sales Manager" and found it already exists with a description that aligns well with the job posting.
2. I then checked the taxonomy for "Managing Partner" and found it was not present.
3. Using the search tool for similar descriptions, I found that the description for "Sales Manager" in the taxonomy had a high similarity score to the description from the job posting.
4. I added "Managing Partner" as a new job title to the taxonomy with a description based on the job posting.
5. Finally, I updated the synonyms for "Sales Manager" in the taxonomy to include "Managing Partner."
The taxonomy now accurately reflects the job titles and descriptions from the job posting.
> Finished chain.The LLM agent is looking at a job posting where both of the titles “Sales Manager” and “Managing Partner” is being used to describe a job. It finds that “Sales Manager” is in the taxonomy, but decides to add “Managing Partner” to it as a synonym. A somewhat questionable decision. It also adds “Managing Partner” to the job title taxonomy, as it is not already present in it.
Finally, as the taxonomy grows, the LLM agent more and more often finds that a job title is already in the taxonomy. Its work then start to center more on ensuring that descriptions are accurate.
> Entering new AgentExecutor chain...
Invoking: `look_up_main_job_title_in_taxonomy` with `Carpenter`
synonyms description
main_job_title
carpenter cut, fabricate, and install wooden and other structures according to specifications. work in diverse settings to produce steady and functional infrastructure and products. requires great hand dexterity, knowledge of wood properties, carpentry materials, and adherence to health and safety standards.
Invoking: `edit_description_of_job_title_in_taxonomy` with `{'main_job_title': 'Carpenter', 'updated_description': 'A Carpenter cuts, fabricates, and installs wooden and other structures according to specifications, working in diverse settings to produce steady and functional infrastructure and products. Requires great hand dexterity, an eye for detail, knowledge of wood properties, and other carpentry materials. Must follow health and safety standards, be able to read technical documents and drawings, and have proficiency in using tools and equipment. Responsibilities include taking measurements, calculating material needs, lifting, and performing various construction and installation tasks.'}`
Previous description for 'carpenter' replaced by updated descriptionThe job title "Carpenter" from the job posting has been found in the taxonomy. The description in the taxonomy has been updated to include the detailed tasks, responsibilities, qualifications, and tools as described in the job posting. The updated description now provides a more comprehensive overview of the role, including the ability to read technical documents and drawings, proficiency in using tools and equipment, and specific construction and installation tasks.
> Finished chain.
[...]
The job title "Carpenter" from the job posting has been found in the taxonomy.
The description in the taxonomy has been updated to include the detailed tasks, responsibilities, qualifications, and tools as described in the job posting.
The updated description now provides a more comprehensive overview of the role, including the ability to read technical documents and drawings, proficiency in using tools and equipment, and specific construction and installation tasks.In the above example, the description of the job title “carpenter” is updated to more accurately reflect the content of carpenter jobs as described in job postings .
The more job postings we run through the LLM agent, the more fully fledged our taxonomy becomes.
There is no limit to how many job postings we can feed to our LLM agent. The more the better. We can also handle other languages than English, if needed.
Caveats
This LLM agent is based on GPT-4 Turbo. Since it is fed large amounts of tokens in order to solve the problem, you can easily run up a large bill. Using a smaller model, e.g. GPT-3.5 Turbo, would require a different setup, as it is less power full and will not solve the tasks as straight forward as GPT-4 Turbo. Further, GPT-4 Turbo still makes questionable decisions at times.
Improvements
If we were to further improve the setup, we could split the overall task into subtasks that are each handled by a dedicated LLM. We could then have an LLM agent orchestrate the overall solving of the task. This would enable us to use simpler models for the subtasks, e.g. GPT-3.5 Turbo, and at the same time achieve a higher quality taxonomy.
Conclusion
We have shown that LLM agents have the potential to automate the building and maintenance of a job title taxonomy from online job postings. But a job title taxonomy is of course just an example of what an LLM agent can do. By giving the right tools to an LLM agent and explaining how it can determine when a task is solved, it is possible to automate tasks that hitherto had to be carried out by a human or improve on existing automations that do not produce results of high enough quality.
