How to measure “inclusion” quantitatively : free resources and research data for D&I advocates
What do you look for when you’re looking for a new job?
Good pay, no commute, better title, opportunities to advance, a rocket ship company… oh, and culture. Culture is important.
Well, I believed (and still do believe) there is a group of people who would put inclusive culture at the top of the list of things they look for in a new job. Because I was one of them. I still believe there are people like me, who would choose a lower paying job over a company run by CE-bros with a toxic culture.
People who want to work at a place where they can bring their whole selves to work, where they don’t have to merely survive while suffering from daily microaggressions and other oppressive BS. Call me naive, but I thought such places can exist, but surely, it would take a lot of research and validation.
So how do you go about finding such a place?
How do you *really* know if a company has an “inclusive culture?”
Well, my business partner and I wanted to find out.
Clue #1: Diversity Reports
In recent years, more and more companies have been eager (and pressured) to share “diversity reports,” sharing their demographic data. Still, the number of companies that have been releasing diversity reports consistently is limited. And, because the format and contents of these reports are so varied, it puts the onus on the job seekers to decipher and compare. Most of these reports breakdown demographic information into race and gender categories, vs. reflecting people with overlapping / intersecting identities, making it difficult to understand the true population of the company.
And as we know, diversity does not guarantee inclusion. In fact, some companies have become better at attracting more diverse talent (“pipeline problem”) that they can quickly replace the folks they lost. The problem with most diversity reports is that they are a snapshot — like a balance sheet — that only gives you a partial picture of reality.
Just like you need a lot more than just a balance sheet to understand a company’s health, you need a whole lot more information than just the current population information to understand a company’s actual culture.
In order to understand the actual culture of a company, through the lens of inclusivity, you need way more than a snapshot of who’s surviving at the company today. But keep those reports coming, we still want to know!
Clue #2: Glassdoor, et al.
Ah, Glassdoor.
I just have three points.
- People don’t trust Glassdoor in general
- It’s hard to find inclusion-specific information on Glassdoor
- See screenshot below
Need I say more?
Clue #3: Company website
HAHAHAHAHAHAHHAHAHHHAHAHAHAHAHAHA
Ok, I do like studying the company’s leadership and board members, though. And their pretty marketing language on how much they value diversity and inclusion. I also think it does count for something (maybe, like, half a sticker) if a company is vocal about valuing D&I vs. not.
Those are just a few resources that are available online. Unfortunately, finding inclusive culture information that is reliable and comprehensive is not easy. Plus, culture varies not only by company, but by team within a company. Getting this data online is near impossible.
Some forward thinking companies have gone ahead and begun the process of measuring inclusion using tools like Culture Amp’s Inclusion Survey, but the information they gather is often kept private and is shared only if the data looks favorable. After all, what incentives do companies have to air their dirty laundry?
So what do people trust?
People trust unaltered feedback from current or former employees of the company. Resourceful people who have access to people at companies they are interested in prefer to talk to current or ex-employees to get the real scoop.
“Tell me the real deal. Should I apply?”
No surprise here, right?
But unless you have this social capital, how are you supposed to access this information? Without access, you have to rely on your own judgment based on limited information available publicly and roll the dice. Just keep a low expectation, is that it?
We wanted to change this.
We wanted to redistribute power to job seekers by providing them with discoverable, reliable, comprehensive, relevant, and comparable data around inclusion.
We wanted to provide a safe platform for employees experiencing toxic or inclusive cultures to speak up. We wanted them to be able to tell their side of the story without the company’s interference.
We wanted to standardize how we talk about and measure “inclusion” and provide a useful tool so companies can improve against concrete metrics. We understand the idea of capturing “inclusion” in one quantitative score may seem simplistic. But what we see over and over again is that businesses don’t prioritize what cannot be measured.
While we would never claim one quantitative metric alone can paint the most accurate and comprehensive picture of a company’s inclusive culture, we did believe it can be a tool used to supplement existing efforts to get a pulse on a company’s culture. Or at least serve as a starting point.
Ultimately, we believed that with better information about companies’ culture, inclusion-seeking job seekers would be able to make better decisions. On the flip side, we hoped this would put pressure on companies to improve their culture to be more inclusive, and measure how they’re doing via real-time, candid feedback from their workforce (most companies we talked to did not like this idea).
So the idea of “inclusion dashboard” was born. We did a lot of research around how one would define and measure inclusion. We interviewed a bunch of people who were exploring new opportunities to understand their research methods and priorities. We talked to D&I experts, companies at the forefront of leading D&I, researchers, professors, Venture Capitalists, and spent months thinking about how we can close this information gap.
We learned a lot.
Here are some interesting, but not surprising, findings / validations from our interviews:
- People who have experienced harmful culture rank inclusive culture higher on the list of priorities when looking for a new job vs. people who have not experienced harm.
- Ensured anonymity was the #1 requirement for making people feel comfortable leaving a company review.
- 4 common factors were identified as a requirement for making information trustworthy: Relevancy of the data to the individual, credibility of the information source, balance of good and bad data (goes back to credibility), connecting directly with an individual who works at the company.
There’s a load more data we gathered that you can check out in our now archived “pitch” deck. Note we never actually wanted to raise capital, we just wanted to get advice from smart people and launch the survey to see where it goes. We had some ideas for how we would monetize, but we first wanted to gather and make inclusion data public and iterate from there.
Anyway, after conducting our own research and interviews, we attempted to develop a way to define and measure inclusion. It contained three major parts:
- Psychological Safety: Do people feel belong and empowered?
- Trust in Leadership: Do people believe their leaders actually care about inclusion? Do they trust their leaders will do the right thing, especially when shit hits the fan? Do they believe their HR team will address complaints swiftly and competently?
- Incidents: Have people experienced, witnessed, or overheard harassment, assault, or other forms of harm based on identity?
We then created a survey to measure this.
A quick note — big props to the team at Culture Amp and Paradigm IQ for releasing the Inclusion Survey — we reviewed their survey and adapted some of the questions (particularly helpful for the Psychological Safety section) and added additional ones that we thought were important for assessing inclusion. We also love all the work Project Include has done — we kept in mind a lot of their recommendations around measuring progress.
We tried to make the survey as concise as possible, especially given the fact that it was intended to be done publicly (aka voluntarily). We needed the process to be as easy as possible.
Our survey contained 30 questions in addition to gathering people’s demographic information, tenure and department at the company, and their level. This would allow us to breakdown the data in meaningful ways, showcasing how different folks at different levels, in different teams, with different identities felt about the company’s culture.
We added important questions like,
- “I feel comfortable going to my HR with complaints (e.g., harassment, bullying, microaggression, etc.)” and
- “I was pressured or incentivized by my company to leave a positive review”
that we don’t usually see in company-run employee engagement surveys.
Here’s the link to receive the full survey.
If you want to adapt it for your company and use it, feel free. Note the survey was still in the process of being polished with help text, definitions, and more thorough answer options.
We were also in the process of finalizing our scoring algorithm based on the survey results and demographic data of the survey respondents. We were toying with some controversial ideas, too, weighting scores more heavily if the responses were submitted by folks in marginalized social identity groups (e.g., people of color, women, queer, trans, disabled, etc.) vs. those in privileged identity groups (e.g., white, straight, man, able-bodied, etc.). This was based on our belief that in order to achieve true inclusion, we need to ensure the most marginalized group feels included.
We were going to launch the survey and gather data publicly, and publish the data in a comprehensive and digestable way using an interactive dashboard. People would be able to slice and dice the data to fit their needs, so if I’m looking for inclusion data from the perspective of queer women of color in Customer Success, for example, then I’d be able to.
We worked with a product manager to scope out the platform dashboard features, prioritized business requirements. For example, we wanted to create a way to verify someone’s identity and employment history via LinkedIn (we know it’s not the perfect way) to solve for credibility, and put a threshold for when we would release a company’s data so we can solve for non-traceability based on demographic and reliability in terms of the data volume (e.g., we wouldn’t release the survey results unless a company had X number of responses, so it’s not easy to trace back who filled it out based on their identity markers).
We looked for advisors. We had partners ready to help us develop the site.
So why did we stop? 3 major challenges surfaced.
- Challenge #1: Gathering a large volume of data to make the site useful
- Challenge #2: Refreshing the data often to ensure it stays up-to-date
- Challenge #3: People don’t make decisions based on information
We believed we could overcome Challenges #1 and 2 eventually. But #3? That was a tough one for us to swallow. What’s the point if we gather and disseminate data but it doesn’t influence people’s decisions? How would we actually influence change? What were we solving for?
“Since the Susan Fowler story broke out, the number of female engineer applicants has not decreased. There also wasn’t a mass exodus from Uber” —very reliable Uber insider
This crushed us. Sure, you can argue “but Uber is different — not every company has the leverage Uber has” or “no, but I really care!”
But for us, it seemed like the ultimate impact of closing this gap was unclear especially when the road to building a useful product was a complex one.
The truth is, unless enough people vote with their employment status, companies are not going to fundamentally shift their internal culture.
It was a tough decision for us to scrap this “Glassdoor for inclusion” idea. We were so passionate about solving this problem, and we still are. We ultimately decided, though, this is not going to be the dream we make reality. We are still using bits and pieces of our learnings today, and have utmost respect for folks trying to tackle this gap.
We don’t regret having done the research.
And we want to share what we learned with anyone who cares. Because we are all in this together, and we have to build solidarity and coalition to move the needle a millimeter at a time.
So we’re open sourcing our materials here:
- Inclusion Score overview deck (modified for public audience) — contains our vision, mission, values, along with research findings and stats we compiled.
- Survey questions
You’ll find flaws in our stuff, no doubt, and maybe even judge us for being naive (but without it, can change really happen?). Basically, please don’t troll us. But do share your thoughts and ideas so we can all learn from each other and be better
The following companies are also trying to bring inclusive culture data to the forefront to help jobseekers. Check them out and support them:
So what are we doing instead? We are providing a new kind of “diversity and inclusion” workshops to develop inclusive leaders. It’s hard work that requires a lot of emotional labor. It’s not very scalable. But we truly believe in the impact we can have on people with our approach, one rooted in social justice and the idea that change happens incrementally and with individuals. It may not be as scalable or sexy as running a software company to some, but we love connecting with real people, holding space for tough conversations, and seeing lightbulbs go off in people’s hearts. We’ve been getting raving reviews and feedback from folks who say we’ve completely changed their perspective on diversity and inclusion. Now that means something.
Have ideas on how to measure inclusion better? Share in comments or message me directly! I’d love to hear from you.
About Michelle Kim
Michelle is an entrepreneur, activist, speaker, and a coach passionate about empowering individuals and organizations to create positive change. She is the co-founder of Awaken and owner of Michelle Kim Consulting.
Follow Michelle’s continued journey to create change in this world:
Subscribe | Website | LinkedIn | Twitter | Facebook | Awaken
