Tech Apprenticeships Series: [Hiring & Sourcing] 3 Key Things to Consider When Hiring Tech Apprentices

TLDR; This article outlines 3 aspects we kept in mind when thinking about hiring and sourcing apprentices for Twilio’s Hatch Apprenticeship Program

1. Apprentice candidates (may) have few connections in the industry, don’t rely on them finding you or referrals — be proactive and target the candidates you want to attract.
2. Coding bootcamp graduate demographics mirror University CS graduate demographics — To attract and hire diverse talent, you must be intentional about sourcing and the process.
3. Evaluate candidates for the work they will be doing (don’t use coding algorithms) — this leads to the best hiring outcome and thriving apprentices.

This is article #4 of the “Tech Apprenticeship Series”

Table of Contents

· Consistent Iteration
· #1 Lack of connections in the tech community
· #2. Hiring bootcamp grads is not a magic DEI solution
∘ Source Intentionally
∘ Reduce barriers to the application process
∘ Reduce bias in candidate evaluation
∘ Interview Training
· #3. Evaluate candidates for skills relevant to the work
· Conclusion

Everything said here has been said before. If you haven’t read Beti Gathegi’s take on sourcing, published in the Kapor Center’s Apprenticeship Playbook, I recommend doing so.

Also note — Twilio’s program required a technical baseline, and while there were some self taught programmers, most completed a coding bootcamp.
This limited the talent pool AND is not representative of what tech apprenticeships can or should be.

Some programs are 0 to 1, meaning the apprentices have no coding experience. We wanted to make longer term investments in partnerships with community organizations and community colleges, but were limited by our capacity.

Consistent Iteration

At Twilio, our Hatch apprenticeship program was always evolving. We sought out feedback from mentors, managers and apprentices:

• 2–3 surveys (application/interviewing, onboarding and exit)
• Frequent listening sessions and “roundtables”

We usually did not get it right the first time, but we always learned from our mistakes (there there were many…). With each cohort, we created a “Reflections and Betterments” document with actionable insights. We then prioritized a few key changes using a participatory ideation session, and went to work.

As we evolved the program, we kept these points in mind to make better decisions. Hopefully they can help future tech apprenticeship programs.

#1. [Most] Apprentice candidates did not have connections in the tech community.

Relying on referrals or word of mouth limits hiring some of the best/most deserving apprentices. Hire with intention and go to the communities you want to attract. Be as clear as you can about the process so candidates know what to expect and how best to prepare.

Quick story — In 2017, I personally bombed an Airbnb interview — I have a diagnosed anxiety disorder, and struggle with interviewing. After this, I asked 3 tech friends to intensely mock interview me. This resulted in me landing offers from Reddit, Instacart, and AdHoc — I was the same person as before, just with practice (and help!) to the specific way tech companies evaluate talent. I was fortunate to have this community, many engineers with high potential may not.

The Problem

Current hiring practices in tech favor candidates that have connections to the tech industry (referrals). This in itself furthers inequity. If you want to find bright, overlooked, and capable engineers, don’t just expect them to find you. Go to them.

As you will read below, bootcamps graduates are not more diverse than CS graduates. It is not an “if you build it they will come” situation. Apprenticeship programs get an insane number of applicants, but in order to hire diverse talent, programs need to put in effort to get applicants from all backgrounds.

Actions to Take

1. Be proactive — Get your programs in front of the communities you want to hire from. See details on how to do this below.
2. Design a hiring process that does not require insider knowledge to be successful. Personally, I don’t think processes that require heavy studying give valuable hiring signals, as it favors those who have resources/people to help them prepare.
3. Be mindful that candidates are new to tech interviews — train interviewers! Interviewers should be supportive, create a relaxed environment (reduce anxiety as much as possible), guide/nudge the candidates when needed, and evaluate for core skills (skills like ability to learn, motivation, communication, ability to take feedback, etc).
4. Over-communicate the process to candidates. Outline the process in writing and provide it upfront. Host open sessions to demystify the process. Consider providing mock interviews for those interested. What does the process look like? What types of interviews will they do? How can they best prepare? Candidates should feel like they know what to expect.

#2. Hiring bootcamp grads is not a magic DEI solution. The demographics of bootcamp alumni is very similar to CS majors. Source intentionally to get a diverse applicant pool.

The demographics are not different for bootcamp grads vs the tech industry. Source: Course Report. NOTE that the US Census data in this infographic is outdated. 2020 data can be found here.

The Problem

The demographics of bootcamp graduates mirror those of CS graduates. Apprenticeships do not automatically get a diverse candidate pool, though it is possible to build one.

Since applicants are not required to have previous experience and there is no (or should be no) pedigree bias (bias given to candidates from “top” schools), there is an opportunity for companies to get really talented, uniquely experienced, driven and diverse engineers. The applicant pool is quite large, and since other companies often overlook non-traditionally trained engineers, there is little competition.

Actions to Take

Be intentional — source intentionally, reduce barriers to the application process, and reduce biases in candidate evaluation. Encourage people to apply if they are considering it.

At Twilio, we —

Sourced Intentionally

1. Did outreach and encouraged applicants from diverse tech organizations, such as Techqueria or /dev/color. (This may be a helpful list 15 Inclusion and Diversity (I&D) Slack Communities to Follow if you Work in Tech)
2. Worked with members from our company ERGs such as Twarriors, Spectrum, Latinx @ Twilio, Women @ Twilio, Black Twilions, to spread the word and encourage people from their networks to apply.
3. Did information sessions with coding bootcamps that support specific communities of developers, such as Hackbright Academy, CodeOp, and Adalab (women focused schools).
4. Partnered with Economic and Workforce Development Organizations (shout out to Orrian Willis at TechSF).
5. Hosted open sessions such as Twilio After Hours where we introduced the company, encouraged applicants to apply, and demystified the process.
6. Held open sessions for applicants to practice interview skills. We had Twilio engineers walk through how to approach coding interviews and best practices for interviewing.

Reduced barriers to the application process

1. We made sure the application was not too long (the application intake was 3 questions) and did not have unnecessary requirements (we tried to keep our requirements very small).
2. Made the program information welcoming and encouraging. You want those that are unsure or on the fence to still apply.

We were not perfect — Originally, we had each applicant submit a 5 minute walk through of code via video just to apply… through feedback, we learned this was limiting applications and was not liked by 50% of applicants. Don’t do that. Make intake easy and focus on evaluating fairly.

Took steps to reduce bias in candidate evaluation

1. We had a hiring rubric for both behavioral (managers) and technical interviews (mentors).
2. We trained interviewers specifically for apprenticeship interviews.
3. Each candidate interviewed with multiple teams to get different perspectives — 2 hiring managers and 2 technical interviewers.
4. Ensured our interviewing team was as diverse as possible.

Interview Training

In our interview training, we were explicit with our interviewers. It was important to us to create a welcoming environment. For technical interviewers, we asked them imitate what it would be like to work together — being collaborative and not evaluative (as in, try to come across more as working together and less as evaluating).

We looked for:

1. Culture add (not fit) — what different perspectives could the candidate bring to the team?
2. “Core skills” (passion for learning, ability to take feedback, communication, motivation/self drive, passion for solving problems, resiliency, collaboration, etc).
3. A technical baseline, though which coding language was less important because we usually taught them a new one from what they knew.

#3. Evaluate candidates for skills relevant to the work they will be doing — this leads to thriving apprentices. Use coding interview questions representative of their future work, and not algorithm questions.

The Problem

The tech industry admits tech interviewing is broken. There are studies to support technical interviewing in its current form (coding challenges) does not give good signal for how a person will perform in the role.

There are only 2 instances where you ever want to use recursion: 1. When you are in a coding interview, and 2. When you want your pull request rejected by your teammates. — Christ Slowe, CTO Reddit

For apprentice candidates, coding challenges that do not represent the real work are detrimental to finding great apprentices, as it misses a core aspect of being an apprentice — learning on the job. It is difficult to evaluate for a technical baseline AND for their ability to learn and take feedback. Candidates that have learned engineering through a bootcamp can demonstrate how quickly they have learned if the problems are tailored to what they learned — practical web application building.

Side Rant — Yes they need a technical baseline, but expect everyone will need to learn, so prioritize learning as a skill. Even if they are really talented with a coding language, they will need to become MUCH BETTER to be a successful “Level 1" engineer and learn a lot of new concepts. The only way they will get there is if they have the motivation, the ability to learn, and ability to take feedback.

Actions to Take

1. Evaluate for potential — no matter what, they will need to learn. Evaluate for their ability to learn, ability to take feedback, growth mindset, motivation, grit, communication, curiosity. In the technical interview, we encouraged interviewers to give feedback/make suggestions to see how candidates responded to the suggestion.
2. Evaluate for what they will be doing in the role (and at their level) — for example, using a directed graph to solve a technical challenge is not relevant, but pulling data from multiple APIs, deduping and aggregating the data is.
3. Try newer ways of evaluating candidates. Have them:
   => Submit code samples and then talk through their design choices in the interview.
   => Build parts of a web app (frontend or backend) and check that requirements are fulfilled, that they test the code, and how they talk about their work.
   => Debug code so you can see how they work through challenges.
4. Index on behavioral interviews. Ask them questions about their experience solving problems, not quitting when things were hard, or taking feedback and growing successfully.

In Conclusion

Hopefully this provides helpful context to a few problems and actions we took at Twilio when thinking about hiring our apprentices with the Hatch Program.

3 important things to remember, in review

1. Apprentice candidates did not have connections in the tech community.
2. The demographics of bootcamp alumni is similar to CS majors, with representation a concern.
3. Evaluate candidates for skills relevant to the work they will be doing — Interviews should evaluate skills that are needed for their future work. Algorithms are not part of it.

Consider

1. Measure the program and iterate. You won’t get it right at first. That is fine, iterate and get better.
2. Be intentional. With everything. Hiring. Sourcing. The interview process. Training interviewers.
3. Learn from others. There are great resources and programs out there already doing great work.

Thanks all!

What other things should be kept in mind when thinking of hiring and sourcing apprentices? Please share if you have any thoughts.