Tech Apprenticeship Series #2: Metrics

Published in

Tech Apprenticeship Series

12 min readJan 19, 2023

TLDR; The apprenticeship at Twilio consistently measured key program health statistics, and many including conversion rate, retention rates and also program satisfaction were higher than expectations and industry averages. These statistics helped prove the the program to be cost effective.

To view the condensed metrics, please see this table at the bottom of the post.

· A High Level Overview Hatch Data
· Top 3 Metrics
· Apprentice Conversion Rate vs Benchmarks
· Demographic Data & Representation
· Retention
· Applications and Interviewing
· Apprenticeship Numbers by Location
· Leadership & Progression
· Cost per Apprentice
· Satisfaction Surveys & Program Feedback
· In Conclusion
· Random Interesting Facts Discovered During Research
· Condensed Metrics
· List of Resources Cited

Apologies for the delay, I have been traveling the past few months, and have also been spending time aggregating the following metrics and stats in order to write this post

NOTE: Any metrics shared here were originally published or shared by the Hatch Apprenticeship Program.

ALSO NOTE: I did my best to research reputable sources to compare the Hatch statistics against, but it was difficult to find good hiring statistics for intern / new graduate software engineers, especially at other top tech companies. If you know of any sources that could be helpful, please send them to me!

A High Level Overview Hatch Data

With the Hatch Program, we measured a lot of things both quantitatively and qualitatively, and used this data to identify “Learnings & Betterments”

From its start in 2017 through 2022, the Software Engineering Apprenticeship at Twilio evolved significantly. We took an apprentice centered, data driven approach to improve our program, with specific focus on apprentice success as engineers, team satisfaction, sourcing, interviewing, curriculum, feeling of support, and apprentice experience.

This post will highlight some of the key successes of the program, many of which challenge the (incorrect) negative assumptions and narratives about the quality of non-Computer Science (CS) Graduate software engineers. Hopefully this data can be part of the conversation demonstrating the great value of apprenticeships and how impactful non-CS engineers can be to an organization.

Top 3 Metrics

We used the following metrics in most of our communications as they were the most important and drove home the value of the program. The end goal was to convert apprentices to full time engineers, and have them stay with Twilio long term.

Total Apprenticeships: 65

Over the course of 5 years (2017–2022), Twilio offered 65 paid apprenticeships.

Apprenticeship Conversion Rate: 91%

Of the 65 total apprentices, 59 (91%) became full time software engineers at Twilio.

Percentage of Apprentices still with Twilio after 2 years: 90%

90% of all apprentices that converted were still with Twilio after 2 years. Many continue to stay at Twilio, having been there for 3, 4 and 5 years now.

Apprentice Conversion Rate vs Benchmarks

At Twilio, Hatch’s conversion offer rate (92%), conversion offer acceptance rate (98%) and overall conversion rate (91%) were higher than our intern conversion rate which was between 70 — 80%. This intern conversion rate is also higher than the industry standard, which I was told is around 60%.

According to a study done by the National Association of Colleges and Employers (NACE), the average 2017 intern conversion offer rate was 67.1%, the average intern acceptance rate was 76.4%, with a total average conversion rate of 51.3%.

NACE also found a correlation between smaller intern cohorts and higher conversion rates, which is an important detail as Hatch’s strong conversion numbers may be tied to the small cohort design of the program.

Hatch Apprenticeship Conversion Rate in Detail

Offered FTE Roles: 60 of the 65 apprentices (92%) were offered conversion to a full time role.
▹ 63 of the 65 total apprentices (97%) earned conversion offers, meaning the managers chose to convert the engineer to full time. Sadly 3 apprentices were not offered due to layoffs.
Accepted Conversion Offers: 59 of 60 apprentices (98%) accepted the full time offer to become an L1 software engineer.
Total Conversion Rate: Of the 65 total apprentices, 59 (91%) became full time software engineers at Twilio.

Twilio Intern Conversion Rate

Without exact numbers, conversations with our talent team revealed Twilio had roughly a 70–80% conversion rate of interns to FTE roles. Additionally, intern and new graduate offers were accepted at an estimated 70–80% rate.

Demographic Data & Representation

While I do not have access to specific demographic data, we did have better than industry average (and general Twilio population) representation of engineers from communities traditionally underrepresented in tech.

70% of Apprentices identified as women or non-binary.

Many publications state that only 20–25% of software engineers are women. The Hatch Apprenticeship had many women applicants, and also strong representation of ~70% women/non-binary engineers in the program. This included hiring in countries where Twilio was struggling to hire early in career women engineers. In one of the countries we operated, 9 of every 10 applicants for engineering roles outside of Hatch were men.

Apprentice Alumni Involvement in Employee Resource Groups

Many alumni were founders and are leaders and members of impactful Twilio Employee Resource Groups (ERGs) such as the Latinx @ Twilio, Black Twilions, Spectrum (LGBTQ+), and Twarriors (Veterans).

Retention

One of the largest benefits of an apprenticeship program, and strong training/onboarding programs in general, is the long term retention of the employees (Harvard Business Review: To Retain New Hires, Spend More Time Onboarding Them).

The Hatch Apprenticeship had high rates of retention, above the industry average. The average turnover rate for the tech industry is between 10–15% (meaning 10–15% leave each year). Hatch Alumni at Twilio had a 5–6% turnover rate per year.

Finding data on retention in the tech industry has been difficult to discover. According to this study using LinkedIn data, the mean employee tenure at reddit is 1.2 years, at zoom it is 1.3 years, and at Meta it is 1.7 years.

PLEASE NOTE that I do not have the non-apprenticeship retention of early in career engineers at Twilio, nor could I find retention information regarding entry level software engineers industry wide — IF YOU HAVE THIS INFO, PLEASE SEND IT OVER AND I WILL INCLUDE IT.

Hatch Software Engineer Retention

~95% of converted apprentices were still with Twilio after 1 year
~90–92% of converted apprentices were still with Twilio after 2 years
~80–85% of converted apprentices were still with Twilio after 3 years

Retention Rates for Interns (NACE, 2021)

Statistics from the National Association of Colleges and Employers. NOTE this includes all industries and all job functions.

After 1 Year

67.7% of employees that interned with their company previously were with the company after 1 year
51.7% of employees that interned at a different company were with their new company after 1 year
35.7% of employees that did not intern at all were with their company after 1 year

After 5 Years

41.5% of employees that interned with their company previously were with the company after 5 years
32.1% of employees that interned at a different company were with their new company after 5 years
26.1% of employees that did not intern at all were with their company after 5 years

Applications and Interviewing

50 Qualified Applicants per Role

We would have 50 qualified applicants (bootcamp graduates, self taught, etc — no CS degree) per 1 role posted.

50% of Candidates passed the ‘Onsite’ Interview

On average, 50% of our candidates invited ‘onsite’ (it was virtual) passed the onsite portion of the interview.

98% of Candidates Accepted Apprentice Offers

Of the offers to join the apprenticeship that we made during my time at Twilio, only one person ever rejected our offer. This made the time spent on interviewing very cost effective, as our offers were almost always accepted.

This compares to an estimated 70–80% acceptance rate for intern offers and new graduate offers at Twilio.

Apprenticeship Numbers by Location

In 2022, the Hatch Program expanded our team and made a few changes to allow us to begin scaling rapidly in a cost effective way. In follow up posts, I will demonstrate how we were scaling and why it would have been cost effective.

Total Apprenticeships (2017–2023)

US Remote — 54 Total Apprentices
Europe Remote — 8 Total Apprentices
India Remote — 3 Total Apprentices*

* We ran a Pilot in India, but paused it after 1 cohort to focus on scaling in the US and Europe, and planned on revisiting that site in the future.

2022 Plans

Our 2022 plans were revised due to the recent changes in hiring strategy.

US Remote — 16 Planned (8 actual apprenticeships offered)
Europe Remote — 14 Planned (6 actual apprenticeships offered)

2023 Plans

Our 2023 cohorts were paused and the support staff was let go due to company layoffs.

US Remote — 24 Planned, 0 given
Europe Remote — 16 Planned, 0 given

Leadership & Progression

A common bias against non-CS degree software engineers is whether they will be able to do the job of a software engineer and excel technically. We found that managers were satisfied with the technical expertise of their apprentice, which is represented in the Manager Feedback Highlights section of this post.

In the 5 years of Hatch, most alumni advanced through the L1 level to L2 roles, and many alumni from early cohorts were promoted to the L3 level (which is a senior level at Twilio).

Many alumni lead and have led complex technical projects, being responsible for the technical design, working collaboratively with PMs, and leading other engineers.

One former apprentice became an engineering manager. Additionally, there are alumni that run engineering-wide guilds such as the “Ministry of Testing” and the “Frontend Knowledge Sharing Forum”.

Cost per Apprentice

The Cost per Apprentice depended on the location, as Twilio has a geo-based compensation policy. If you are interested in ballpark figures, reach out to me on LinkedIn and I am happy to provide them.

On principle, apprentices were paid the same salary as interns and given full benefits such as 401k match, health insurance, wireless allowance, book reimbursement, and wellness benefit.

If you are going to do an apprenticeship, pay the apprentices. Otherwise you are being exploitative.

The cost of a US based 6 month apprenticeship for Twilio was roughly $60,000 — $75,000 per apprentice including salary, benefits and the cost of the support team. This varied due to cohort size and location. The cost of an L1 engineer also varied by location, but could be in the US it varied from $125,000 to $150,000 including salary and benefits (which are estimated at 20% of the yearly base salary). This shows that 6 months of an apprentice with the support team was similar to 6 months of a direct L1 hire.

This may seem like a lot, but with Twilio’s conversion and retention of the apprentices, and the work they were able to accomplish, the benefits of an apprenticeship outweighed the cost. It was as cost effective as hiring interns or new graduates.

Apprentices were able to earn a competitive paycheck while Twilio received great, long term talent. When factoring in the cost of the support team, apprentice salaries, but then also conversion rates and long term retention, apprentices cost less than a direct L1 hire.

Converted Apprentices’ Salary Info

When apprentices converted, they were paid the same amount as other L1 software engineers as they were doing the same work as they were. They even had a 4 month head-start with onboarding.

I strongly advise to not have a separate track for converted apprentices (as I have seen at some companies) compared to other engineering talent. This persists inequality, unfairly biases the apprentices as less technical in the eyes of managers, and is not fair to the skills, work produced and value that apprentices bring to their teams.

Apprentices typically brought to their team a strong technical foundation, 4 months of onboarding, and high professional maturity and strong core skills, such as project management, communication, and problem solving.

Satisfaction Surveys & Program Feedback

We typically surveyed the apprentices, mentors and managers at least 2 times — once after hiring and the first 2 months of onboarding, and once as an exit survey after the cohort completed. This information was then used to create a ‘Learnings and Betterments’ document and helped us complete our cohort retrospective. We typically then identified projects to focus on improving the program before the next cohort began.

I have included highlights (positive feedback) from our surveys given between 2020–2022. I will include critical feedback in upcoming posts.

Apprentice Feedback Highlights

94% of Apprentices were satisfied with their experience as an apprentice at Twilio, and would recommend the program to a peer.
94% of Apprentices found the feedback/performance review process helpful, a process in which they received 3 separate rubric based evaluations over the course of the apprenticeship.
88% of Apprentices were satisfied with the application and interview process to the apprenticeship, which included small written answers (such as “what interests you in joining the apprenticeship?”, “what skills from your former profession do you think will help you as a software engineer?”), a simple baseline coding exercise, and pairing with a Twilio engineer for the onsite interview.

Manager Feedback Highlights

92% of all managers would recommend the program to a peer. We had many repeat managers that were happy with their former apprentices and returned as they saw the program as a strong source of engineering talent.
100% of all managers of converted apprentices were optimistic of the growth of their converted apprentice as an engineer at Twilio.
When comparing the technical contributions of their apprentice vs their expectations:
▹ 54% of managers rated it above their expectations.
▹ 31% of managers rated it as meeting their expectations.
▹ 15% of managers rated it below their expectations.
When comparing the non-technical contributions (communication, project management, contributions in meetings, organizing opportunities to connect socially, driving a culture of communication, etc) of their apprentice vs their expectations:
▹ 62% of managers rated it above their expectations.
▹ 31% of managers rated it as meeting their expectations.
▹ 7% of managers rated it below their expectations.
93% of managers found the program’s performance feedback structure for apprentices helpful.

Mentor Feedback Highlights

93% of all mentors would recommend the program to their peers.
100% of mentors of converted apprentices were optimistic of the growth of their converted apprentice as an engineer at Twilio.

In Conclusion

Metrics and statistics only paint one part of the picture, but the goal of sharing these is to demonstrate the potential of non-CS, early in career software engineers. As you can see by the amount of applicants, the high conversion rate, the long term retention, and the examples of impact and progression, the Hatch program did not experience the perceived risks often attributed to hiring bootcamp and self taught engineers.

The main takeaway in our experience is that if you have a great team of managers and mentors, with a bit of intentionality, non-CS engineers can flourish and produce a lot of value for any engineering organization. And if you feel like you do not have the right managers and mentors in your organization to support non-CS early in career engineers, you should probably re-evaluate who you are hiring in the first place.

Random Interesting Facts Discovered During Research

These are some interesting facts discovered while researching this post.

A 2021 LinkedIn Learning Report showed that 94% of surveyed employees would stay longer if their company simply invested in their learning.
27% of Gen Z expect to work for their first full-time employer for 2 years or less (Working for The Future: A Report on Recruiting and Retaining Gen Z).
Recruiting employees into mentorship positions increases internal engagement and retention chances by 50% and collaborative relationships by 37% (Human Resources Director: Retain talent through workforce development programs, originally cited in Internships: An innovative employee retention strategy for the post-COVID work world).
On average, an employer spends 33% of an employee’s yearly salary on their leave and replacement (19 Employee Retention Statistics that will Surprise You, (2022), originally cited in Internships: An innovative employee retention strategy for the post-COVID work world).