Introducing Sentropy
Hate speech, harassment, and abuse are more visible across the web than ever before. We’re introducing Sentropy and talking about the tech we’ve built to help protect the internet.
Disclaimer: the text below includes obscene and offensive language.
Introduction
Taylor Rhyne: Welcome to Sentropy! This is the very first official blog post that we’re putting together. Should we say what we’re doing?
John Redgrave: We’re doing an audio blog!
Taylor Rhyne: And why did we decide to do an audio blog instead of writing a blog?
John Redgrave: We chose to do an audio blog because we felt like the nature of the problem we’re working to solve is such that you really want to get the emotional elements, and frankly writing a blog post, you just can’t pick up on the emotion and the passion that we have for this problem. And so we thought telling you what we’re doing is going to be a much better medium than just writing it down.
We’re going to tell you a little bit about the company, why we started it. But in order to do that, we thought it’d be really cool to have one of our newest employees actually ask us questions about what we’re building, why we’re building it, and the nature of the problem we’re solving.
Taylor Rhyne: Let’s do a quick round of intros and John, let’s start with you and then I’ll end by introducing the person that’s going to take us forward from here.
John Redgrave: Sounds good. So I’m John Redgrave. I’m the co-founder and CEO of Sentropy.
Taylor Rhyne: My name is Taylor Rhyne. I’m the COO and one of the co-founders of Sentropy. And we are joined by Nix.
Nix, do you want to introduce yourself?
Nix Maasdorp: Yeah, sure. So I’m Nix Maasdorp. I’m just graduating from Harvard Business School and I’m really excited to be joining the Sentropy team and leading the Product Marketing effort.
Taylor Rhyne: And where were you before HBS?
Nix Maasdorp: Before HBS, I was at Dropbox for five years where I was working on a bunch of different teams in the beginning in the early days, and then focused on Product Marketing and the B2B space.
Taylor Rhyne: And I should point out for everybody, that Nix has not actually officially started at Sentropy yet, and she agreed to do this even though we haven’t actually started signing her paychecks, which was incredibly generous of her to do.
So with that, we’re gonna hand the mic over to Nix and let her just start asking some questions to learn a little bit more about Sentropy and give John and me a chance to talk about what it is that we’re building as a team.
What we’re building
Nix Maasdorp: Great. For the folks out there that have never heard of this company, what is it that you’re building?
John Redgrave: We’re building Sentropy to fight abuse online. This is a huge problem. It’s deeply important to me, not only because I’m a father of two young kids, but also because I’ve watched the ramifications of hate, harassment and other forms of abuse, and that the impact that they’ve had on my friends, my family.
I unfortunately watched the ramifications of this on friends who grew up in Florida and went to Marjorie Stoneman Douglas, as well as seeing a very close family friend who was shot 15 times in a mass shooting. It just felt like it was time for more effort to be poured into this problem. And at the end of the day, we know some amazing people who could actually make a dent in this problem together. And so hence, we started Sentropy.
Taylor Rhyne: I’ve always found it easiest to talk about Sentropy as a content moderation company, at least in the position that we’re in today. If you take content moderation, you can split it up into two different categories: there’s the data side and there’s the more functional role, what it’s like to work as a content moderator. Neither of those is particularly attractive.
On the data side, we are going after the content that everybody else in the world wants to avoid. There’s physical violence, there’s drugs, there’s sexual services being sold online, things like racism, sexism, identity attacks. That’s the kind of stuff that we’re actively going after, and we have to, because it’s important for us to see the different ways that that type of content shows up in real conversations in real communities across the web so that we can model it and then use those models to go out and help our customer base detect it.
I don’t think that anybody would look at that side of this problem and say, “oh, that’s really interesting, I want to go and throw myself into that.” But I think the functional job of being a moderator is even less appealing. When you look at the media coverage of this industry, you see a lot of stories about the moderators who are sitting in front of a computer for seven or eight hours a day and all they’re doing is clicking, row by row or image by image to review content that’s been reported by users. That’s a very psychologically taxing and mentally challenging job to do day-in and day-out. So you get this huge number of people that are doing this all day long around the world and the burnout rate in that industry is huge.
This is the world that Sentropy is stepping into, and what we think that we can do is build detection capabilities to allow our enterprise customers to identify where is there real abuse, where is there real harassment, what types of malicious content are impacting the user experience that they’re trying to create? And by helping them detect and identify that, we can basically show them where the needles are in their haystack and let them focus on extracting those needles so that they create a safer, more enjoyable community like what they originally intended.
John Redgrave: This isn’t about replacing humans. This is about creating the “Iron Man suit” for content moderation. A content moderator, if they’re moving as fast as humanly possible, is going to be able to review roughly 3,000 pieces of content a day. And when you think about the magnitude of these platforms, platforms that, you know, don’t deal in thousands, they deal in millions or billions of pieces of content or transactions between people a day. It’s just insufficient that a human moderator is going through content on a row by row basis or on a piece-by-piece basis. It’s really important that the content moderation world has the best technology and tooling to increase the efficacy of the job that they’re doing.
The problem
Nix Maasdorp: And from what I’ve seen, this is a big problem, as you say, that many big companies are trying to solve and are spending a lot of money on in the process. Can you speak a little more on that?
John Redgrave: The reality of this problem is it has a bottom line impact for these companies. So it’s not just the pure amount of money they’re spending on this problem, but why do they care? Why are they choosing to spend money in the first place?
There’s a really clear correlation between users who experience harassment and churn. We’ve seen that about 13% of adults stop using platforms after seeing harassment of another user and the number jumps pretty considerably up to 30% if they’ve actually experienced harassment themselves. So when platforms think about this problem. This is not just about keeping your users safe. This is actually about protecting your business.
That is a disconnect in the space today. It’s almost like what cyber security used to be. The cyber security industry used to be filled with companies that hand-rolled their own tooling, they didn’t share information amongst each other, and it was actually quite a small spend area in terms of resource allocation in the companies. Now, you look at companies today, almost all of whom have CSOs and spend billions as an industry as a whole to protect their businesses.
But the concept of cybersecurity has such a disconnect from the concept of security in the physical world. When you talk to another human being about security, what do they talk about? Sure, they talk about their passwords and their bank account information, but they actually talk about their safety. And that is really problematic in that companies, and especially these large brands, have under-invested in the concept of safety because they haven’t fully grokked that if you protect your users, you actually are protecting your business as a whole.
A lot of companies have relied so heavily on the users to be on the frontline of this problem, but research suggests that upwards of 75% of harassment goes unreported. That’s a huge problem! And so how do you understand it better? You understand it by looking at data. But where do you get the data? Well, you have to have really robust detection systems in place, and the companies who are investing on this problem — like Facebook, like Google — they’re so much better equipped than they used to be to handle this problem, and yet they still have a long way to go. That’s where we think Sentropy plugs in is to bridge the gap.
Taylor Rhyne: The big companies like Facebook and Google and Twitter — they have two things that every other company in the world does not have. They have resources and they have resource flexibility. And so Facebook and Google can throw a ton of money into hiring contract moderators that are located around the world and can work on this problem for a very low hourly wage. That is one way to scale.
Beyond just having a huge stockpile of cash, they also have resource flexibility. They can take engineers that they have internally, redirect them towards building internal systems that understand natural language and understand how people behave on their platform to help them make this a more tractable problem within their individual products and communities.
Those two things are resources that almost no other company besides Facebook and Google has available to them, and you really need both at the same time in order for this to be something that you can make progress against. If you take a community that has ten million monthly active users or a million monthly active users, you’re talking about a community online that is a very small fraction of the size of Facebook or Google, but they still have a tremendous amount of content that they’ve got to review.
They can’t throw humans at this problem the same way that those big companies do, and they almost certainly don’t have the talent internally to think about the different natural language issues and how to solve this from a technical perspective. If they do have that talent internally, it’s usually not something that they’re able to allocate towards content moderation, because those engineers are very rare, they’re very valuable, and they’re more often than not working on the core product, not on Trust & Safety issues, which is usually regarded as kind of a back office function for smaller platforms.
Why it’s challenging
Nix Maasdorp: So for these big companies that have the internal resources and resource flexibility that you’re talking about — why haven’t they been able to solve this problem? What makes it so tough?
Taylor Rhyne: Yeah, this is a great segue from the previous question. On the one hand, we’ve been talking about all the resources that Facebook and Google have to solve this problem, but on the other hand, you can just open up one of their platforms and you won’t have to do a bunch of digging to find examples of abuse or offensive content.
Now, the fact that even Google and Facebook haven’t solved this problem is, to us, it’s a testament of how challenging this is to do at scale. You could go and pick a random person off the sidewalk and tell them that and I think they’re going to interpret it as pretty surprising. We as humans have this incredible ability to take all of these different layers to a conversation or an interaction and understand context and intent by combining them all together and interpreting what’s actually being said. We don’t even give it a second thought that we’re doing that; we’re just good at it. But for a machine that’s very hard to do.
Take the word “bitch.” Now when I say that, I’m guessing there’s some part of you that registers that as an attack, but then you walk away from me saying it, knowing that I’m not actually attacking you. I didn’t say it at you; I said it in a conversation around the word. I gave you an indication that it was coming. In other words, you’re taking a bunch of contextual signal that you have about me and about the conversation that we’re having right now, and you’re using that to inform your interpretation and your reception of that word.
Now, if I say, “go kill yourself, bitch,” then suddenly it takes on a very negative connotation, even if I’m doing it with a nice tone of my voice. The flip side of that is if I say something like, “happy birthday, bitch.” That is a very common colloquial usage of that word these days, but it totally changes the context of that word and how it’s being received by the person on the other end of this conversation.
So you need to know something about who you’re interacting with, and how that word or phrase is being used in a conversation before you can make a smart prediction about whether this is a hate term or racial slur or an identity attack, or if it’s something else entirely.
This industry is pretty littered with companies that have tried to fix this problem by building a better list of hate terms, but if you take a longer list , you try to deploy that without having contextual understanding baked into that approach, you’re going to end up detecting a lot of things that aren’t actually harassment, what we’d call false positives. The flip side of that is equally important to consider. If you’re building a list of hate terms, you’re only going to be able to catch the things that you’ve actually encoded into that list. So again, if you’re not using context, you’re going to end up missing the things that are only hateful or abusive because of the context that they have in that interaction. Those are false negatives, and we need a system that can minimize both the number of false positives and the number of false negatives so that the things that we’re spending time reviewing as human moderators are actually the ones where we should be spending time looking at them, the things that are truly abusive or harassing. But we also want to make sure that we’re not missing a bunch because we haven’t built out a big enough dictionary of bad words or slurs.
All of that requires context and acquiring context is incredibly difficult to do at scale. That’s the real answer to your question. That’s why this is so tough, because first you need to understand what’s being said, but then you also need to understand who is saying it, who’s on the receiving end of it, and where that message is being posted or where that interaction is taking place.
Sentropy’s approach
Nix Maasdorp: So how would you describe Sentropy’s approach to fixing all of this?
Taylor Rhyne: The literal answer your question is that we have two products that help serve different parts of the market. We’re building technology that can help both very small platforms that have never thought about content moderation before and might have nothing in place to do this today, all the way up to very large platforms — communities that have tens or hundreds of millions of users that are extremely active and very conversational.
In order to serve such a wide diversity of customers we found it very important to have different form factors to access the technology that we’re building. The first product that we built is called Sentropy Detect. That’s an API that receives content from a customer and it returns back the different categories of abuse and harassment that we’ve detected with the confidence that we have associated with that label. You can think about this as a programmatic solution that allows larger customers to easily integrate the cutting edge natural language technology that we’ve built into existing workflows.
Once we had that product in place, it became very easy for us to wrap additional UIs around that API to make it easier for the data that gets submitted to the API from our customers and the data that we returned back to them easier to get processed and used by our customer base.
The product that we’re just getting ready to release the market, our second product, is called Sentropy Defend. That’s a browser-based UI that allows those of our customers that might not have any content moderation workflow or tooling in place today to pick up a tool that’s incredibly easy to use. We built it to be very intuitive and it lets customers do end-to-end moderation, so they don’t have to worry about how to build out the detection system in the first place or think through how to actually implement actions on content that they deemed to be against their policies.
They can instead open up our platform, see all of the content that we’ve detected to be abusive or harassing according to our taxonomy and definitions from their community, and then they can use the UI that we’ve built to actually take actions to remove content, ban or suspend users, or execute whatever actions they decided to implement during the configuration of Defend.
John Redgrave: We built a system that allows companies to detect abuse as well as to defend their users and themselves against abuse. The analogy that we give to people is, you know, as Stripe did for financial payments and Twilio did for communications, we are doing for abuse detection. We believed that that type of model is what the market needs. It’s easy to access, it fundamentally allows people to alter the landscape and create a healthier conversation and a healthier community. Healthy means less user churn, higher retention, and more investment into that platform by that community, both in terms of energy, time, and dollars.
Taylor Rhyne: Stripe and Twilio and other developer tools out there that follow a similar business model like Zendesk and PagerDuty — all of these companies are incredibly successful today. They’ve built these fantastic products that the entire tech industry raves about, but really the powerful feature of the businesses that they built was that they took a mission critical element of running an online platform that was not core to the product that was being built, and they said, look, this thing is really difficult to build and maintain. Instead of trying to do this over and over again every time you start a company or you’re trying to scale a company, we’re going to productize this entire functional group and we’re going to turn it into a tool that’s really easy to use. You can sign up for it and in five minutes you can start using it in production.
That has been done for a bunch of different pieces of the tech stack. But content moderation and Trust & Safety have not been touched by that approach yet. And so we think that the business opportunity for Sentropy is to do exactly that. We can go in and say, look, this entire group and this effort that you’ve had to invest in for years and years and years to do content moderation and Trust & Safety no longer needs to be built from the ground up every time. We can build a tool that gives you access to all of the technologies that you need in order to do this in a first-class way, but you don’t have to build them yourself. We can build all of the pieces of that and you just have to worry about how do you actually do the investigation and the moderation that you need in order to protect your user base.
Our advantage
Nix Maasdorp: Your description of how Sentropy is approaching this is different to how other companies are approaching it. While they built great products, you’re talking about solving something that’s never been solved before in a new way. You’re not talking about building a better version of something that already exists.
Taylor Rhyne: That is a good point. We are trying to solve something that has never been solved before. Where those other companies that we used as an analogy on the business model side had really significant problems that they had to go out and solve, they were solving things where there was a known solution, it was just hard to build. People knew how to process payments before Stripe came along, it was just difficult to do that yourself and you never wanted to deal with the pain of building that system.
That’s not the case for us. It’s not that there is technology out there that can go out and figure out when somebody is using the b-word as a form of endearment because they’re friends with the person they’re talking to, or if they’re using that as a way to attack somebody because they’re upset with the way that they played in the last Call of Duty game or because they have some beef with each other at school.
That is what we have to build. And so, yes, there is a significant challenge that presents itself to anybody that comes up and decides to start a content moderation company. I think that we are uniquely positioned to solve that problem precisely because of the team that we’ve built at Sentropy.
This is not something that you can go out and just hire any engineer to solve. You need people who have extremely deep expertise in, not just machine learning, but natural language understanding. And if you look at the whole population of people that actually can make production quality progress against that problem, you’re talking about, you know, maybe dozens or hundreds of people in the world that have ever done anything like that. And I think that we have some of the best people in the world who have worked on that problem working at Sentropy.
John Redgrave: One of the advantages that we have at Sentropy is a number of us worked together at a previous company called Lattice Data. We sold the company to Apple, and we were doing some really interesting work with Apple to help evolve Siri. But I think the most interesting work that we did at the company was with DARPA through the Memex program. The Memex program was targeted at fighting human trafficking, and the human trafficking domain is really complicated. In particular, we were looking at the signals that you would find in places like Craigslist and Backpage and dark web forums. And in these places, the type of behavior that you see is that there’s language that people would use that would be commonplace associated with human trafficking.
That’s a really complicated machine learning task that we were insanely good at. And so it’s that type of behavior and that type of technology that you need to be able to think about the problem of content moderation and more broadly, of abuse online and the ability to actually detect abuse with really high precision and recall. We know exactly how to solve this problem and to make sure that any company that’s engaged with us is actually ahead of the curve.
In this space, you have an adaptive adversary problem. The adaptive adversary is constantly looking for ways to get around the moderation systems, so you need to be able to keep up with them at a minimum. the only way you can do that is to deploy the most cutting-edge machine learning techniques to identify how language is evolving in these ecosystems, where it’s evolving from, and be able to use that knowledge broadly across ecosystems. That’s one of the benefits that we can provide through Sentropy.
Taylor Rhyne: What we’ve been able to build so far internally is impressive even to us. We’ve beaten our own expectations of how quickly we could make progress on the machine learning part of this problem. We’re able to measure that progress by tracking specific metrics that help us understand how often were right and how much of the data that’s out there that we want to be able to catch we’re actually able to catch with the models that we deploy to our customers.
At the same time that we’ve made progress on the raw machine learning power that we’re building, we’ve also built in bias awareness . This is really important, and it’s not something that a lot of machine learning startups are capable of doing just because it takes an added layer of attention to the data that we have that trains and tests our models.
This is doubly important for our customers because while it’s possible for a customer to pick up some open source machine learning library and code up their own classification algorithm, it’s very hard for them to maintain and monitor that in production over time unless they have engineers that are dedicated to that problem.
That’s one of the benefits that we offer, not just that we’re monitoring where we’re right and where we’re wrong and making constant improvements to the quality of our models, but now we’re actually monitoring for bias as well. So our customers can start to rest assured that the moderation technologies they’re using aren’t unfairly evaluating different types of content or user populations within their community.
The impact
Nix Maasdorp: So given Sentropy’s approach, it sounds like, in my understanding, some tangible outcomes are that Sentropy’s able to proactively detect harmful behavior before needing to receive a user report that that behavior has existed. And also you’re able to defend against new forms of abuse that haven’t ever been seen before. Would those be two accurate outcomes?
John Redgrave: Ya, exactly. Astute pickup, Nix. We definitely think about the value prop as multi-fold, but one way to view our technology is we are-helping people to identify abuse more proactively. We don’t believe that users should be on the very frontline of this problem, which is what essentially the companies who rely solely on user reports are doing. Secondarily, we have the capacity — once user reports come in on once things are flagged — to be able to help people to move through a queue very rapidly and to make a much higher volume of decisions and make those decisions more consistently against their stated policies, as well as more efficiently.
While we have continuously evolving global models that are learning from the open web, the gray web, the dark web, as well as our customer data that we can extract back as features without any sort of privacy implications, we also are learning about the specifics of communities, so instead of automatically moderating, we actually get to see the types of decisions that are made, and we believe that those decisions will be world-class out of the box and will allow people to make better decisions, but also as we watch people make decisions, it allows us to make more relevant predictions for the companies that we’re working with.
It’s a really nuanced distinction, but I think important because as you go out into this space, what you find is most companies don’t believe that a single model or another company’s model is going to work perfectly for them. And instead of building something that is hyper-customized or a full-blown services business, we’ve instead built this in as a product feature such that companies can interact with the output of our models through Detect. Then through the Defend product, they can show us how they work, they can show us the types of decisions that they make, they can show us where they disagree with our models, and our models become more relevant based on that type of interaction.
Nix Maasdorp: That makes a lot of sense. So every company that you’d be working with is obviously dealing with unique situations. So something that might be shared on a recipe site, they would be looking to stop words that are very different to words that they’d be trying to stop on a gaming site. So you’re saying that it’s not a plug-and-play for everyone. You do have incredible technology that’s available to all the different types of companies, but you will have a feedback cycle and learning cycle to make sure that you can adapt and have it be customized to that specific company.
Taylor Rhyne: That’s right. So if we started working with a cooking website and a video game company on the same day and both of them sent us a message that was posted on the respective platforms that had the phrase “go kill yourself”, then on day one of using Sentropy, they’re both going to get a label back from us that says, this looks like physical violence. The cooking website will probably agree with that. Unless you’ve got some sort of fringe alt-right cooking site that I’m not aware of, I think that most people would agree that “go kill yourself” is a form of physical violence, so it would be likely that they would agree with that label.
The video game website, however, might say, look , this is a conversation about in-game characters in League of Legends . This is not a conversation where somebody is attacking somebody else. It’s a conversation where they’re talking about the characters in the video game killing themselves, and so we disagree with this label. Those are both equally valuable decisions for us to see feedback on. In the first instance, where we’re working with a cooking website, they will be able to confirm that the linguistic and behavioral features that we use to make that prediction were correct. But for the other customer in the video game world, they’re going to tell us, look, this label was wrong. And so our models will actually learn in real time that the features and weights that we used to make that prediction on Customer B’s platform should change the next time that we see a phrase like “go kill yourself.”
As you compound decisions like that and the feedback that they generate over time, both of those customers end up with essentially customized models that have learned what that platform really cares about. What do they moderate very frequently? What types of language, behavior, and content do they often take action against? And what do they consider to be okay within the normative behaviors of their community?
What’s next
Nix Maasdorp: So when will we be hearing more? What’s happening next for Sentropy?
John Redgrave: That’s a great question. We never believed that when we started this company, we’d be able to get six months down the line and launch. We’re building a complicated machine learning system that requires advanced natural language understanding. We’re now at the point where we’ve been building for almost two years. We’ve been finishing up some private betas of both our detect and defend products. And we’re about to launch to the public very soon, which is really exciting, not only for me and Taylor, but for the entire company. People have poured their hearts and souls into what’s been built here.
I think people have come to Sentropy not only because of the technical merit of the problem, but people deeply care about the mission of giving back to the people the internet that they deserve. It’s really important and we’re really excited to be able to get our company out into the world, but also to just be more vocal about this problem space. We need more strong voices who are standing up for moderators to get better tech, to, frankly, stand behind users, to push for more user safety on these platforms, and we’re excited to be that voice.
You’ll see something in terms of a launch in June. And for the people who are interested in hearing more and learning about the launch or chatting with us, there’s a way to sign up on our website, sentropy.com to get that information and to get in contact with us.
