A Career in Trust & Safety: You know more than you know

Part 1 of the Social Science to Trust & Safety Series

7 min readJun 4, 2022

“I don’t have the skills needed for industry research… Industry research is boring”

Near the end of their studies, many social science graduates ask themselves, what’s next? Do I continue on this academic path or explore an industry research career? Each person I’ve met who is contemplating this decision has their own personal reasons for the direction they ultimately choose, but if you’re considering industry research, this article is for you.

As someone who made the leap from academia to industry and has since spoken to many people contemplating doing so, I have noticed two common misconceptions: 1) believing that social science graduates do not possess the necessary skills to apply for an industry role, and 2) that industry research is basic and boring. I’m here to tell you that neither is true and especially that you know more than you know!

The career path that I want to focus on is a career in Trust and Safety, also commonly referred to as Integrity. This career track not only specifically mentions social science education as a preference in job advertisements but also involves highly stimulating and complex research, debunking both of the aforementioned misconceptions. Unfortunately, awareness of this career track among social science graduates appears to be low, despite the demand for this exact talent profile.

In writing this article, my hope is that more social science graduates will become aware of Trust and Safety as a career option, no longer see industry research as an intellectual compromise, and be able to visualize how their experience can directly plug into the product development process.

What is Trust & Safety?

A quick content analysis of job advertisements listed on Google that mention Trust and Safety or Integrity (“trust and safety” OR “integrity” AND “job”) captures what it is fairly well, with some variation between companies. Across listings, the description of this field includes:

Identifying the safety needs and pain points of users
Developing policies that dictate what behavior is and isn’t allowed within the product
Understanding how to prevent abuse and harm to users
Proactively identifying emerging user safety risks
Understanding how to build trust with users and society

The reality of product development is that the intended use of a product is rarely the only way in which it is used in the world, and sometimes what is developed for good can also be exploited to cause harm. For instance, one person’s social support forum can be another’s scam directory. It is the goal of Trust and Safety teams to set the rules and prevent, detect, and combat the harmful use of products.

To be as concrete as possible, a good way to understand the world of Trust and Safety is to review existing product policies or community guidelines. Current policies or guidelines can tell you a lot about the types of problems that companies are grappling with and may give you a glimpse into the research projects that you might lead should you work there. Here are a few examples to springboard your search:

“Hmm you’ve got my attention, but why am I right for Trust & Safety?”

Trust & Safety ♡ Social Scientists

What does it mean to feel safe? What is trust? How do we measure trust? How should we define harm? Which behaviors are harmful and which are not? What deters people from breaking rules? When harm occurs, how do we remediate it?

To effectively answer these questions, you need to understand people and society. Thus, the fundamentals of social science are critical for the development of well-functioning and legitimate Trust and Safety systems.

Contrary to the common misconception that social science graduates have little to offer within industry, Trust and Safety teams both need their methodological training and theoretical knowledge. The former, methodological training, is commonly written about so I won’t go into detail on it here. In general, research roles within industry often use the same methods trained in social science degrees — ethnographic field observations, qualitative interviewing, trend and sentiment analyses, surveys, behavioral experiments, text analyses, etc (see this article for a more in-depth overview). If you’ve been trained in social science research methods, you likely already have the fundamental skills needed to conduct Trust and Safety research. With respect to the latter, theoretical knowledge, Trust and Safety teams need researchers with good intuition for how people think and behave both individually and en-masse.

“Interesting. What would my role in Trust & Safety look like?”

In conversations with graduate students and academics considering a move to industry, one of the most common questions I’m asked is “what does an average day look like for a researcher in trust and safety?” To which my answer is often, there isn’t an average day and it varies by team and company. Instead, one way to visualize your role is to understand how it fits into the product or policy lifecycle. Let’s look at the example of account security and password behavior as a relatively straightforward, illustrative case that can apply to several online products.

Sample Case: Password Strength

Context: Imagine you work for a company that is building an online product. For this product, users are required to create an account from which they can access the product offering (e.g., movies on Netflix, listings on Airbnb, sale items on Amazon, etc). In this case, the Trust and Safety team may be responsible for building the account security features to keep users’ personal and financial information safe from unauthorized access (i.e., hackers). The main account security feature that your team has built is a username and password login system.

Imagine that, shortly after you join the team, the data scientist notices a trend of spammy user behavior which also happens to coincide with an increase in customer support requests. You review a random sample of these requests and discuss the data anomaly with your data scientist. Together you determine that several user accounts have been compromised. To understand the root cause of the problem, you might partner with a security analyst or investigator. Imagine the investigator determines that a common pattern across the compromised user accounts is the use of a weak password.

Problem Statement: People are using weak passwords when creating their accounts, leaving their accounts vulnerable to compromise.

Research Request: This is an example of a user problem, that is, something about the way users are behaving is problematic for the product system. To understand how to remediate the situation, the team needs to understand a) why users choose weak passwords when creating their accounts and b) what can be done to prevent compromise via weak password use.

Research Execution: Given the relevance of this problem for almost all online products, a researcher may choose to combine several sources of information. You may choose to combine findings from a) a market analysis reviewing the account setup process of other online products, b) a literature review of published academic and industry research on the topic, and c) interviews with some of the users of your product who have been hacked. Together, these data sources would help you to understand why people use weak passwords and what solution options may be feasible.

Product Ideation: As a social scientist, you may already intuitively know why account security systems like passwords are suboptimal. Several theories of human cognition and behavior may already tell you a lot about why people might use weak passwords. For instance, you may understand that people use similar heuristics to create passwords for ease of memorability (e.g., DOB1234 or childnamechildDOB) or that people commonly underestimate their own risk of compromise (Optimism Bias or Invulnerability Illusion).

Based on the results of your research and your social science intuition, your next step might be to partner with a Product Designer to turn what you’ve learned about users into feasible product solutions. You may partner with your Product Manager to decide the features of a successful product (e.g., ease of use, market familiarity, solvability, business cost, build time, etc) and rank solution options, deciding on one or more ideas to experiment with. In some cases, you may conduct further user testing to assess how users react to the proposed solutions or you may conduct a “Black Mirror” brainstorming exercise to try to anticipate any unintended consequences of the proposed solution prior to testing.

Experimentation and Launch: Let’s say that together with your team, you’ve selected a preferred solution option and you’re ready to test it out. This is the all-hands-on-deck phase where you may partner with your Product Designer and Software Engineer to build the new product, and partner with your Data Scientist to decide on the experiment design (e.g., sample size, test market, length of the experiment, analysis plan, success definition, etc).

Prior to launching the experiment, you may conduct a final audit of the product to make sure everything is working properly across different devices and languages, if applicable. Following the completion of the experiment, you may work with your Data Scientist to analyze and interpret the results and make a launch/don’t launch recommendation for the product team. Should the experiment show positive results, your work on this particular project may be complete, however, should the results be inconclusive or negative, the team would likely request more research to understand what to try next.

Research and the Product/Policy Lifecycle

“OK, I can really see myself doing that! So I guess I do want to explore a career in Trust & Safety, where do I start?”

Up Next

If you’ve made the decision to apply for a research role in Trust and Safety, the next step is to identify which research role is best for you, because there are many! Are you a Design Researcher, Policy Researcher, Market Researcher, UX Researcher, or Research Scientist? In part 2 of this series, I will write about the different types of research roles within the Trust and Safety industry, how to know which one would be the best fit for you, and why it matters.