Why Interviewing Prospective Hires is Futile, and How to make it (only slightly) Less Futile
Evergreen is a collection of links to the best learning resources in business, collected by a group of managers, founders, and investors. We contribute resources about one topic, which are synthesized and shared in this Collection. The goal is to learn more efficiently through increased context and focus.
Remember, these are designed to feel like short books, you’re meant to meander and spend ~3 hours on this topic. Save some of these links and read them throughout the week. Immerse yourself in this topic and leave the week smarter than you started it!
We’re back with a new Edition about Job Interviews. Here’s what you get in this post:
- Why Job Interviews are (Nearly) Useless
- Manage a Good Job Interview Process
- What To Not Do (What not to do? Idk which is right)
- What to Look for When Interviewing
- A Shitload of Great Interview Questions
Skip around as you please — enjoy and please add any great content or concepts you think are missing.
Why Interviews are [Nearly] Useless
The first interesting fact that I learned researching how to do job interviews well is that they are nearly useless. This is actually proven with academic studies.
I didn’t know this when I chose it as a topic, but here we are. We’re all going to keep doing them anyway so we may as well eke out all the benefit that we can from them.
Here’s why interviews have so many flaws.
The Observer Effect
The act of observing something will influence the thing being observed. This absolutely seems like the right place to start. The Observer Effect turns an interview into a professional lying session. What I hear in a typical interview:
Interviewer: “Welcome! Please lie to me about your relevant work experience?”
Employee: “[Extrapolates, expands, and exaggerates]”
Employee: “Now, please lie to me about a typical day at your company?”
Interviewer: “[Skips, skims, and smoothes over].”
This seems unlikely to create a strong lasting relationship. It is like getting married after the first few dates, when everyone is only sharing their most generous assessments of themselves.
The *Utter* Uselessness of Job Interviews
According to the NY Times, job interviews aren’t just useless, they are *utterly* useless. Citing studies they had done on whether interviews where additive in a GPA prediction challenge:
In the end, our subjects’ G.P.A. predictions were significantly more accurate for the students they did not meet. The interviews had been counterproductive.
Research that my colleagues and I have conducted shows that the problem with interviews is worse than irrelevance: They can be harmful, undercutting the impact of other, more valuable information about interviewees.
To go even farther to illustrate the point, they also conducted a round of the study where some prospects, unknown to the interviewer, were to give answers determined at random.
Strikingly, not one interviewer reported noticing that he or she was conducting a random interview.
Granted, maybe these people sucked at interviewing, but it does show the weaknesses of the interview structure rather well.
The key psychological insight here is that people have no trouble turning any information into a coherent narrative. People can’t help seeing signals, even in noise.
The Job Interview Process
No matter your process, have a plan. Have it down on paper so your whole team is aware, and you can communicate clearly with your prospect.
Division of Labor
Evaluating prospects is a nuanced skill. It requires practice. Managers who are hiring only a few positions per year are NOT coming from a good place to evaluate talent. They have no sense of base rates.
Chet Cadieux, the CEO and legendary operator of QuikTrip, has a great take on Interviewing. He has dedicated full-time interviewers. Not just an HR screening call, but full-time expert hirers for every position in QuikTrip, down to cashiers in convenience stores.
QuikTrip is a case study in genius operators using simple principles. To read more about QuikTrip, check out Intelligent Fanatics, a wonderful book about some of the world’s best operators.
And thanks to Sean Iddings for contributing Chet’s talk!
My friend Jonathan Howard sent in a post by VC Josh Hannah about Extreme Reference-Checking. It was the only post submitted that talked about it in any detail, and I think it completely nailed it. Everything I’d wish for in a post about Reference Checks.
Here are the gems:
Resumes, CVs and interviews are all sales pitches. The candidates have carefully crafted the attributes they show you in order to seem perfectly suited to the job.
Base 80 percent of your hiring decision on thorough reference checking
But not the way you’re currently doing reference checks… because you’re probably doing them wrong.
In my experience, the leading cause of executive hiring mistakes can be traced to waiting to make (perfunctory) reference checks until the final step in the hiring process, perhaps even after the decision has been made.
You call those references with a confirmatory bias, practically begging them to tell you that the interviewee is a great marketer, so you can get her on board and get back to work.
If you’re not making these calls seeking truth, you shouldn’t even bother. Be in the mindset to dig for negativity, take any answer as an opportunity to drop a prospect.
Expect at least a 50 percent chance of eliminating the candidate based on references. If you are not in this mindset, you will only hear what you want to — that you’ve found the perfect person.
Keep that confirmation bias out of your reference checks — do them early, and do them thoroughly. Read the rest of Josh’s post, there was too much great stuff to excerpt all of it.
Tryouts are better than Interviews
Get as close as you can to a real work environment with a prospect. Contract project, Shadow day, a few sales calls, research sample, etc.
Matt Mullenweg is known for an extreme implementation of this:
The most significant shift we’ve made is requiring every final candidate to work with us for three to eight weeks on a contract basis. Candidates do real tasks alongside the people they would actually be working with if they had the job. They can work at night or on weekends, so they don’t have to leave their current jobs; most spend 10 to 20 hours a week working with Automattic, although that’s flexible.
The goal is not to have them finish a product or do a set amount of work; it is to allow us to quickly and efficiently assess whether this would be a mutually beneficial relationship. They can size up Automattic while we evaluate them.
It’s not always possible to do this scale of “tryout”, but the more you can stretch for, the better.
What Not To Do
As always, we try to learn by inversion. Not just what to do, but we take the steps to find and avoid common pitfalls. Avoiding stupidity is usually easier than achieving brilliance.
I had the confidence to tell hiring managers and executives that putting people in the “maybe” pile was likely to give the candidate a bad experience. “Maybe” really means “no for now.” Better to tell a candidate that you don’t have a role for them right now so they can put their energy into finding a job somewhere else. If something changes, you will be in touch, but for now the park is closed.
Toughen up, buttercup. You want to run a good process for prospects? Don’t leave them hanging. (This has the added benefit of forcing decisions and keeping your hiring funnel clean and clear.)
Don’t Talk Too Much
I once witnessed someone conduct a 90-minute interview in which they talked at least 80% of the time. This interviewer decided that they loved the candidate and hired the prospect on the spot. He was fired in less than two weeks. Not a happy ending for anyone.
Roger Toor has corresponding advice:
Don’t fall into the trap a lot of interviewers do, of making it about you. All too often I see interviewers asking questions to show off how smart they are or that they happened upon something obscure. Or they end up talking a lot about themselves.
We’re all smart and like to show off a bit. An interview provides an imbalance of power uncommon in everyday life, and it can be tempting to use that to bask in our own egos for a while. But that is not going to help you build the right team. Shake it off.
The Other Mistakes
What to Look For When Interviewing
Ok — Here’s the heart of this Edition. The good stuff on what to look for, and how to find it.
Have a Blueprint
It seems like most companies have a loose idea of what they want in a hire, and a set of questions… but few are truly connected. An interview process should be seeking confirming and disconfirming evidence. Having a rubric-type document to work through the process seems the best way to stay organized, impartial, and intellectually honest.
Find Proof of Internal Motivation
Is there proof of internal motivation or external motivation?
Does their resume focus on credentials and titles (inputs), or results (outputs)?
Do they talk about high status people or places they worked, or about the actual work they did?
Why do they do what they do?
Find Proof of Ethics and Character
Are they a good person? How do I know?
Do they do the right thing, even when they don’t have to? Why?
Do they speak truth to power?
Do they try to bullshit you at any point in the interview?
Pay attention to the little things — Do they interrupt me a lot? Do they respect me? Do they respect all people, e.g. how did they treat Rick (our driver) when he picked them up from the airport?
Having all of these “proof points” written out, and as a structured guide to the interviewer reduces the dangers of confirmation bias, liking bias, narrative bias… It keeps things more organized, more effective, and more honest.
Keeping this sheet updated throughout the process, by different interviewers can help keep candidates straight, and ensure your process gets deep, rather than 4 different people conducting the same surface-level interview.
Trajectory, Not State
It is much easier to assess a snapshot in time (state) than correctly predict a pattern (trajectory). I can tell you the score at half-time, for sure. I’m less certain who is going to win the game.
Be sure you are evaluating a candidate’s trajectory, not their current state. When you’re going to have a long and happy working relationship with someone, you want to know they are growing with you, not in stasis or declining.
This short post, Trajectory Matters More Than State, covers this dynamic in candidates and companies. Pay attention to trajectory and do your best to understand what this candidate might be like in months or years.
Thanks to Vinish Garg for submitting this!
Look for those “Soft Skills”
If you’ve ever worked with a talented jerk, you have an appreciation for the importance of soft skills. Internal Communication and coordination are incredibly important to the function of a company, and these are doomed if your team lacks soft skills.
This post by Seth Godin talks about how you can make a plan for assessing soft skills, and how to look for them.
Wisdom — Have you learned things that are difficult to glean from a textbook or a manual? Experience is how we become adults.
Perception — Do you have the experience and the practice to see the world clearly? Seeing things before others have to point them out.
Influence — Have you developed the skills needed to persuade others to take action? Charisma is just one form of this skill.
There are many more examples in the post, and you’ll need to develop your own set of desired attributes for each position.
Finding Attributes through Questionnaires
Matt Frost shared this post by Adam Grant: “What’s wrong with job interviews, and how to fix them.” Grant covers all the ways Interviews miss the mark, and has a novel suggestion for solving them.
The most effective questions are called situational judgment questions. Instead of asking candidates to describe how they handled a unique situation in a previous job or organization, it’s more fruitful to describe consistent situations that candidates could face in this job or organization, and ask them what they would do — or how they would reason.
That’s not the novel part. That’s just the setup for this:
Interestingly, these situational questions don’t require interviews at all; they can be completed and scored online. Researchers have validated situational tests to assess applicant characteristics as diverse as integrity, personal initiative, emotional intelligence, and aggression. In one study with Dane Barnes of Optimize Hire, we gave salespeople a situational questionnaire about their tendencies to take initiative, and then tracked their annual revenue. Here’s what we found:
Low initiative: $53,798
Moderate initiative: $118,808
High initiative: $155,663
If this works, it should mean big things for hiring — knowing what to look for, and having scalable, cheap online tests that look for them is a dream come true for companies. Of course, it means honest cooperation from applicants.
I’m super interested to see how processes like these find their ways into hiring over the next few years. It seems like pure hubris that has kept more objective research from entering the hiring process thus far.
Why Google Loves Structured Interviews
Google has some of the highest standards for its employees of any company today. Solving the challenge of how to keep this bar high across every hiring manager in the company has some challenges:
In other words, most interviews are a waste of time because 99.4 percent of the time is spent trying to confirm whatever impression the interviewer formed in the first ten seconds.
The goal of our interview process is to predict how candidates will perform once they join the team. We achieve that goal by doing what the science says: combining behavioral and situational structured interviews with assessments of cognitive ability, conscientiousness, and leadership.
To help interviewers, we’ve developed an internal tool called qDroid, where an interviewer picks the job they are screening for, checks the attributes they want to test, and is emailed an interview guide with questions designed to predict performance for that job.
This makes it easy for interviewers to find and ask great interview questions. Interviewers can also share the document with others on the interview panel so everyone can collaborate to assess the candidate from all perspectives.
The article is excellent, but it is just the start — get the book, Work Rules! by Laszlo Bock for the full Google hiring playbook.
Here’s a Shitload of Great Interview Questions
As promised — all of the best questions and collections of questions we could find for you. There’s a lot of them so I kept the context short.
Ask “Why? Why? Why? Why?”
Until they start crying or just walk out. Kidding. This post shows you how to use “Why?” to get past the interview-level bullshitting and see what people are actually made of.
Thanks to Vignesh Srinivasan for contributing.
How to ask about a candidate’s “Grit”
Grit (determination) is a key attribute for success in most roles. But you can’t just ask if someone is a determined person. Here is how to ask questions that will show you the secret of Grit.
One Question to ask to find the top 5%
“Did you feel at your last job that you were appropriately appreciated?”
Read (slightly) more in Shane’s short post.
Vinod Khosla’s Giant List of Questions
This post also from Vinish Garg, thanks Vinish!
Another Giant, Comprehensive Hiring Guide
There’s a ton of info in this “How to Hire The Right Person” post from NYT Business, from questions to the setting in which to ask them. Too much great stuff to excerpt, but a valuable resource for those about to hire.
What to Ask and What it Tells you
We couldn’t make it through an Edition of Evergreen without linking to some fantastic post on First Round Review, and here it is. Excellent Questions about surfacing soft skills, and interpreting candidate answers that aren’t always easy to parse.
The Importance of Great Follow-Ups
Lastly, let’s look at Book in A Box again. Tucker’s company doesn’t just have a great rubric to work against, they keep a great bank of questions to have on hand. Importantly, they ALSO plan follow-up questions, which is where the good stuff really comes out.
Here’s a sample:
“Tell me about your all-time biggest professional screw-up.”
Wait for their full answer. THEN follow up after they tell you about it, and ask: “What did you do about it?”
“Tell me about the best idea you’ve ever had that your boss/collaborators/client didn’t go for. The idea that got away.”
Wait for their full answer. THEN follow up after they tell you about it, and ask: “What did you do when they said no?”
“What’s your favorite thing or activity to do, work or non-work related? Why?”
Wait for their full answer. THEN follow up after they tell you about it, and ask: “What have you started or initiated in that area?” If nothing, ask why not.
And to close out, here’s my favorite question off any of these lists:
“If we don’t hire you, why do you think that would be?”
There are a ton of opinions about how to do job interviews. I think this huge variety is possible because of how little any kind of interview seems to impact the ability to get the right person in the seat.
At the end of all of this research, I came away thinking: “It seems like everything else AROUND the interview actually matters much more.”
Things that matter more than your Interviewing Skills:
- Your company’s “brand” to employees
- recruiting strength of the organization
- available talent pool, the marketing of the position
- ability to “sell” prospective hires
- employee onboarding
- company culture shapes how employees work
Sure, try to do it well. But don’t obsess over perfecting it. Work harder on skills with higher returns.
If you enjoyed this, you’ll get more like it by joining Evergreen:
My Other Projects
Upcoming Book: The Almanack of Naval Ravikant
Mini-book: Career Advice for Uniquely Ambitious People
Evergreen Lives on Donations
Creating Evergreen takes a lot of time and decent amount of money. If you find anything helpful and interesting here, please consider buying me some BBQ ($11 Z-Man & Fries at Joe’s KC), or really anything between a coffee and a good dinner. Thank you very much! ❤ ☺
Massive appreciation for those who suggested pieces of content (or wrote something new) for this Edition of Evergreen: Tucker Max, Sean Iddings, Shane Mac, Itamar Goldminz, Vignesh Srinivasan, Kathryn O’Day, Matt Frost, Vinish Garg, Roger Toor, Edson Rigonatti, Jonathan Howard, Nick Seguin, Ben Center, Dominic Bouchard, Peter Fenyo Fenyvesi, Kenny Fraser, Stephane Gosselin, and Erik Martin.
Many thanks for being a part of this project! Not every suggestion is able to make it to the final edit, but every single suggestion is read and appreciated.
As my Father always says: “There’s always room for the best.” There’s always a better resource out there. These collections can always get better, and I hope that they do. If you can think of anything that was missed, I welcome you to share it.
If you liked this, check out other Editions of Evergreen:Building and Managing a Team:How to Find and Recruit the Team you Need
How Not to Hire like a Clownshow
Compensation Rules Everything Around Me
Why Employee Onboarding is holding you back
How to Boost Employee Retention
How Performance Reviews are being Reinvented
Secrets to Perfecting Organizational Communication
How to Manage Scale, and Operate in Scaling Organizations
How to Fire an Employee
What you actually need to know about Company Culture
How to Interview Prospective HiresStrategy and Competitive Advantage:How to Master the Craft of Strategy
Competitive Advantage: How to Build a Winning Business
The Power of Network Effects
How Cost Leadership Builds Powerful Businesses
Why the Best Brands Stand Out
Scale as Competitive Advantage
Barriers to Entry are Confusing
Flywheel Effect: Meta-Competitive AdvantageBuilding the Business:How to get good business Ideas: Mental Alchemy of Ideation
How to Choose the Right Business Ideas
Product/Market Fit: What it really means & How to Measure it
How to Failure-proof your business with Customer Development
How Strategy and Psychology Work Together to Perfect Pricing
The Most Important Equations in Business - CAC (Part 1)
The Simple Math Behind Every Profitable Business - CLV (Part 2)
How Psychology behind Word-of-Mouth Works
The Secret Core of Every Successful Business--Distribution
The Most Important Lessons in Sales
Why Value Creation is the Foundation of Business
Why Value Capture is the most important idea you haven't read about
The Misunderstood and Underestimated Genius of AdvertisingHow to be a Great Human:How to Start a New Job: Handling Career Transitions like a Boss
How to Master the Discipline of Product Management
The Ancient Origins of Storytelling, and how to Apply ThemI've also written about How & Why we started Evergreen:How a prototype's failure created the next iteration
Mission & Method of EvergreenFollow me on Twitter: @EricJorgensonAnd Please... Join Evergreen.