Rank and Review
Definition
Rank and Review enables us to Staff with Stars by objectively identifying top performers, highlighting where team members need coaching, and pinpointing people who need to go to bootcamp. Once you’ve had an opportunity to apply the frameworks within a team, you will notice a nifty progression within the frameworks: enforcing the quality bar provides the objective data you need for a Rank and Review which then shows you how you need to Shrink To Grow.
A great rank and review has three components: a clear stack rank, unit-specific coaching, and data-driven decisions. First, a successful rank and review clearly stack ranks the team by quality (internal FTAR). Very quickly, you should be able to say who has the best quality and who has the worst quality. Next, a successful rank and review has specific coaching for each team member that equips them to improve and holds them individually accountable. Lastly, the best rank and reviews lead to clear and data-driven decisions on which team members need coaching, are producing the right level of quality, and need to go to bootcamp.
Step 1: Stack rank by quality and record dispersion
The first thing you need to do for rank and review is stack rank your team by quality (external FTAR). You can do a Rank and Review based either on quality or the team’s metric depending on your current priorities (see note 2 at end of this chapter). Since all teams in the factory need to improve their quality, we’re going to proceed assuming you’re doing your Rank and Review based on quality.
This should be easy to do and not take long. If this is not the case, you’ll need to read the Enforce Quality Bar chapter before proceeding with Rank and Review. Since the stack rank serves as the basis for evaluation, team members should be crystal clear on a) what is expected of them and b) where they stand at all times. Once you’ve done your stack rank, you want to observe “dispersion”: the difference in performance from the highest to the lowest. If dispersion is 3x, you have an issue (see shrink to grow). Teams should not have more than a 1.5x dispersion.
Step 2: Unit-specific coaching
Once your team is stack ranked by quality, it becomes very clear how to coach each team member. Imagine a certain team member has a 76% internal FTAR. This means 24% of their units are failing the quality bar and need to be improved. If each team member produces 25 units per week, you have 6 individual units or opportunities to coach this team member.
Most managers who aren’t aligned with our culture provide irregular and high-level coaching. Things like: “You’re doing a good job but you should pay more attention to details.” We have two guiding principles for coaching that will enable you to create a great rank and review. First, every piece of coaching needs to be tied to a specific unit of work (which should be linked in your deliverable).
Second, every coaching point needs to hold team members accountable — it should give them something specific to change/improve. Here’s a great example of coaching: “Configure your coverage tool to catch these cases of switches without default statement, and also casts prone to null pointer exceptions. Have the discipline to apply your coverage tool to both production and also Unit Test code. Case1 Case2”
Step 3: Data-driven categorization decisions
One of the most valuable outcomes from rank and review is clear categorization for every team member and decisions for each category. After step 1 and 2, you need to put every IC into one of three categories (which is achieved by drawing the green and red line of your deliverable):
1. Meets necessary quality bar
2. Coach (not high enough quality but less than a week of check in chat coaching will fix)
3. Send to bootcamp (significantly below the quality and not worth coaching)
These categorizations need to be data-driven. The first group can be calculated simply by deciding what the acceptable quality level. For the second group, once you have closely reviewed each team member’s units, you should be keenly aware of what they need to do to improve quality and therefore the number of hours you need to coach them. Repeat the same analysis for the third group and if it will take longer than a week, it will be more efficient to send them to bootcamp.
Step 4: Set expectations for the particular skills you expect from Engineering Bootcamp
Many managers mistakenly use ‘send to Bootcamp’ as a catch-all for their bottom performers. These managers think to themselves: “Well, this guy is no good so I’ll just get rid of him by sending him to Bootcamp.” Managers also often have incorrect expectations about what Engineering Bootcamp does and doesn’t teach.
If you have an IC who is consistently producing low-quality units because they do not have deep technical knowledge about fringe case Linux scenarios, sending them to Bootcamp is not going to fix the problem. If you have an IC who can’t get out of their ‘productivity-first’ approach with less than 2 weeks of your coaching, Engineering Bootcamp will fix the problem. To learn in detail what Engineering Bootcamp teaches, please read the ‘What is the WSPro Framework Matrix’. Additionally, here is the Engineering Bootcamp playbook for reference.
When you are doing your Rank and Review, you do not want to automatically put your low performers in the ‘Send to Bootcamp’ group. You need to do first do deep dive on why this IC is a low performer. Once you have the root cause, decide if Engineering Bootcamp will address this root cause.
If it will, great! If it won’t, you should not send them to Engineering Bootcamp. To ensure Engineering Bootcamp is the right spot for your IC, you should identify 2 specific skills you expect your IC to learn while in Engineering Bootcamp that will fix the root issue. The complete list of skills is available in the ‘What is the WSPro Framework Matrix’ chapter. As you make ‘Send to Bootcamp’ decisions in your slides, explicitly mention the 2 skills that, once learned, will fix the root issue of your IC.
When Bootcamp is not the answer: letting people go
Occasionally you may run into a scenario where an IC does not fall into any of these 3 categories. Perhaps an IC simply doesn’t have the skills necessary for the role and neither your coaching nor Engineering Bootcamp will address the issue. These scenarios should be rare but when you encounter them, don’t fall into the temptation to send the IC to bootcamp. The right answer is to let the IC go in with a ‘Mother Teresa’ touch.
To be fair to the IC and fix the underlying problem the first thing to do is document the skill gaps. What are the specific skills that this person does not have that the other ICs have? Once you have this defined list, double check and ensure it can’t be fixed by a reasonable (max of 2 hours/week for a month) amount of coaching or Engineering Bootcamp. Once you have identified the specific skill gaps, it’s important to take this list to your VP and the Crossover marketplace team to decide how the recruitment process needs to change to avoid hiring more people like this. It may also be wise to do a deep dive on how this person was hired in the first place.
Staff with Stars is a core value that determines how we hire and manage talent. We have historical data to prove that an understaffed team of stars (A-players) is far superior to a full team of B-players. WSPro is about paying higher salaries, getting the best people and delivering the lowest cost via higher productivity. Rank and Review is the central process we use to achieve Staff with Stars.
Important Notes:
1. Some managers mistakenly think that team members with the highest quality do not need coaching. The best ICs are always seeking continuous improvement and it’s your job as a manager to equip each team member with opportunities for growth. Instead of putting an artificial ceiling over top performers, you want to identify what’s holding top performers back from the next level and help them get there.
2. You can do a Rank and Review with either the team’s metric or the external FTAR. Both of these produce an ‘objective’ Rank and Review because they are based on objective numbers. As a manager, you need to decide which type of Rank and Review to deploy. Since the entire factory needs to improve quality, your Rank and Review should be based on external FTAR (or internal FTAR if you are in Manager Bootcamp and meet the criteria).
3. Objective vs. subjective Rank and Reviews: You may find yourself in a situation where you look at the objective stack rank and are uncomfortable — it doesn’t match your perception. Let’s imagine you are convinced person A is by far your top performer but in the objective Rank and Review, they are in the bottom third of the stack rank. First, you should fight the urge to tweak your metric (or external FTAR) to ‘force’ your objective Rank and Review to match your subjective stack rank. This would be jerry-rigging your Rank and Review which you don’t want to do. Instead, have a discussion with your VP on why this is. Ask ‘5 why’s’. This will be a productive discussion for you and your VP.
4. If you are in Manager Bootcamp and your team has a poor external quality bar, you can use your internal quality bar as the basis for your Rank and Review. The IQB component for this point reads: “Ranking should be based 100% by external FTAR. However, if the manager decides the existing internal FTAR or newly proposed internal FTAR produces a more accurate quality stack rank, the manager can stack rank their team this way. In this case, a link to the rationale behind this decision needs to be linked in the slide”
Examples: link
