Ava — a GPT for Engineering Managers (Part 2)
Using GPT to simulate an engineering management interview
👋 Hi, this is Nitin with another post in my series The Engineering Manager’s Pocket Guide. In every post, I cover topics related to tech and leadership through the lens of an engineering manager. This post is part of a 2-part series where I share an experiment with creating a custom GPT for EMs. To get all the posts, subscribe here.
In Part 1 of this series, I introduced Ava, a custom GPT assistant I built to help with my work as an Engineering Manager (EM). The main question I wanted to answer was: “Can I create an assistant that helps manage some of my EM tasks?” Toward the end, I asked Ava to generate a sample Q4 roadmap, and to my surprise, it did a pretty solid job.
Role-playing with Ava
With Ava up and running, I started thinking about practical ways to use it in my day-to-day work. Lately, I’ve been focused on hiring, so I shared some technical interview tips (signal over noise) with Ava. That got me curious about whether “Ava could play a bigger role in interviews?”, which became the motivation for this post.
For this experiment, I decided to do some interview role-playing. The setup was simple: Ava played the role of an interviewer in one chat and an EM candidate in another.
Here’s what I needed to make it work:
- A few types of interviews (I chose behavioral, system design, and project retrospectives).
- A set of questions for each interview type.
- A rubric or scorecard to evaluate the responses.
Create questions and scorecards
To solve the first step, I asked Ava to generate a list of questions it might ask an EM candidate, along with corresponding scorecards. I also had it produce these as PDF files.
Note: I’ve truncated the screenshots for brevity.
Here’s how the scorecards were created:
An example scorecard:
Act as Ava, the candidate
Next, I had Ava switch roles and act as the candidate. I fed it the questions and asked for responses, which it provided (again in a PDF). As expected, the answers were fairly generic — not exactly what you’d get from a real 1-hour interview discussion.
Example answer for Ava-candidate
Act as Ava, the hiring manager
Finally, I gave Ava the scorecards and the answers and asked it to rate the responses. It did a decent job evaluating itself, though unsurprisingly, it rated itself pretty highly.
It felt like running a real interview, but with a virtual assistant handling the bulk of the work.
Looking ahead, I’d like to fine-tune the model with more question/answer training data to raise its bar for quality, and make it better at grading responses. There’s a lot of potential here, and this is just the beginning. ☮️
