On the selection and hiring of Intelligence Analysts (Intel Shop in a Box — Part 1)
There’s been a great deal of ink spilled about improving the quality of intelligence over the decades. Since 2001, we’ve seen a dramatic increase in the number of state and local agencies which have adopted intelligence analysis as some part of their mission (DHS sponsored fusion centers being but one type)and with them, the number of personnel assigned to conduct that analysis. Most efforts to improve the quality of analysis tends to focus on training. That makes a certain sense when you think about it. Agencies are trying to improve existing capabilities and to paraphrase one former senior governmental official ‘You do analysis with the analysts you have, not the analysts you might want or wish to have at a later time.’
It doesn’t hurt to have a ton of private companies pushing analytical training as the solution to the problems that have been identified over, and over and over again. I’ve done my fair share of teaching these types of courses over the past five years and while they have real value I can’t help feeling that we’re missing a vital step. In the dozens of courses I’ve taught I’ve met some highly motivated and very intelligence persons who were clearly well suited to the profession of intelligence analysis. I’ve also met huge numbers of people who left me scratching my head and thinking ‘Why did the person in charge of hiring ever think this person would be a good fit for this position?’
I recall hearing discussions around the 2006–2008 time frame in which it was hypothesized that the influx of new, talented people exposed to new, improved training and refined procedures would lead to a renaissance of intelligence analysis once they got a few years under their collective belt. Think of it as a ‘trickle up’ theory of improvement. I remain unconvinced that’s going to happen for many reasons but for now I’d like to focus on one: The selection and hiring criteria for analysts.
Hiring processes for intelligence analysts are virtually no different from any other position. A combination of who in your social network can put your resume in the right hands and a whole lot of cognitive bias. We generally make a decision quickly, based on a bullet point on a resume or a firm handshake in an interview and riff questions in the interview looking to reinforce those beliefs. Let’s face it, if you’ve been in the job market for any length of time you see these interviews a mile away and have a pretty good idea of how to manipulate the interviewer.
Last year I had an opportunity to put together a selection process for an intelligence shop and how, a little more than a year — and six hires — later I think it’s time to declare victory. Here’s the process:
Specifically identify what skills you’d like to see in your candidate. For an analyst that might include things like ability to write/speak clearly, organizational skills, experience working in a joint agency environment, etc. Then outline a small scale of expertise to rate each skill. For example, if you consider language skills desirable you could do something like:
- 0 points - no foreign language skills
- 1 point - ability to read/write or speak a non-desired language with near fluency
- 2 points - ability to read/write/speak a non-desired language fluently
- 3 points - ability to read/write or speak a desired (native to Middle East/North Africa)language with near fluency
- 4 points - ability to read/write/speak a desired language (native to Middle East/North Africa) fluently
Additional language ability is associated with all sorts of positive benefits so the skill gets the candidate some credit. If it’s a language that’s actually needed for the position it’s a double whammy and so gets additional points.
Don’t overthink this too much. The point is to provide a better, more systematic way to review resumes over just eyeballing them or comparing them to a generic job description.
Gather all resumes and ‘sanitize’ them of personally identifying information. There’s a growing body of evidence that unconscious bias has a significant impact upon the hiring process. Therefore, we wanted to take out any reference to gender or race that we can as early as possible. Make sure to create some sort of key so you can figure out which resume belongs to whom. I labeled each resume with a number, copied the resumes and then sanitized one of the copies. It was then an easy task to find who belonged to #42.
Hand off the sanitized resumes to someone else and have them score the resumes against the skill sheet you created. I had a couple of analysts do this part and would recommend they score them separately and then compare their totals. If you don’t have the time or personnel to do that, having one person score the resumes will work as well. You will now have in front of you a stack of resumes, scored according to your own criteria.
Review your top candidates (match up the sanitized resumes with the complete ones) and exclude any that are inappropriate for further consideration. In my review I excluded candidates who lived more than one days travel away (There were so many qualified candidates from the surrounding geographical region that we decided to eliminate the added element of complexity or coordinating travel schedules) and were seriously over-qualified (A former general officer applying for an entry level analyst position probably isn’t going to be a good fit). These exclusions should be very few in number (out of 200 applicants I excluded only two candidates) and your review should be a quick one looking for obvious disqualifiers. If you find you’re excluding more than obvious outliers you should either reevaluate your scoring criteria or have someone else review the resumes.
Borrowing shamelessly from Daniel Kahneman, I went thought about an ideal intelligence analyst. What characteristics would she have? If I was going to describe this person, how would I do it? Quick-thinking? Thorough? Works well under pressure? Persuasive speaker? These may be the same as the skills we identified in the first step but these may include characteristics that don’t lend themselves well to being ferreted out in a resume. Select 4–6 key characteristics and then create 4–6 questions that specifically address those characteristics.
Call in your top candidates for an interview. You’re hiring for analysts so have them demonstrate some analysis. Require all candidates to write a brief (2–4 page) original piece of analysis and deliver the findings in a 3–5 minute briefing. I wasn’t too specific in topics for analysis, asking for anything related to the field of terrorism, homeland security or intelligence analysis. Their specific topic selection also gave some insight into their thinking process. Some analysts did try to present recycled work (generally a disqualifier unless they were able to demonstrate an original component to the work). While no one selected for interviews dropped out, we also considered this process as important for weeding out ‘time-wasters’. Writing and preparing an original product and briefing takes time and it becomes pretty obvious who rushed, who cut and pasted out of Wikipedia and who put the work in. If they can’t put forth maximum effort for an interview, they’re unlikely to do so once they’ve got the gig.
Interviews should be done with several people (we found four to be a good number). They started with the presentation of the original analytical product and some brief Q&A about it. Then, we went through the scripted questions. Everyone had the questions in front of them and were asked to independently score the answers on a 3 point scale (good, fair, poor).
I should note that by this time in the process I had not looked at the resumes in any detail (only a cursory check to look for obvious exclusionary traits noted above). As a result I was able to view the candidates ‘fresh’ without any preconceived notions about what to expect. Two other people on the panel had reviewed resumes and were able to ask specific questions related to that. My original intent (again based on a system Daniel Kahneman describes in his book) was that we wouldn’t even need to ask resume based questions. However, there were a couple of instances where they did provide real insight. So, my recommendation would be to include them but only use them for context in order to get better insight into the qualities and skills you’ve identified as essential to a high functioning analyst.
After the interview, I compiled the scores from the resume evaluation and the interview, looking for any huge discrepancies between the rankings. In some cases the people who had clearly padded their resume were easy to identify as they did not score well in the interview. I was surprised at just how much the scoring criteria focused the minds of all the panel members and resulted in near unanimous agreement on the top candidates. The scores, the briefing and the written product were then used as ‘evidence’ of the candidates suitability for the position.
One year on, what can we say about the process? My answer is going to be inherently subjective. We still are struggling to identify reliable quality metrics for analytical products let along analysts. What I can say is that after 25 years in the field (lo, where has the 8th Guards Army gone?) I have not worked with a more uniformly high-quality team. The process was sufficiently different (and difficult) for the candidates that they now consider it almost as a rite of passage that not only binds them together but set the tone for high standards when they came on board.
In terms of diversity — important not only as a ‘feel good’ value to pursue but also because a diverse team conveys important benefits to a team of analysts — we did pretty well. Of six hires, five were female and two were veterans. While not on our radar at all, our candidates represented a very broad range of religious affiliations which has led to some interesting discussions and insights I didn’t anticipate. I was surprised to see a lack of ethnic diversity in the candidates (all six candidates would be classified as ‘white’ by the US census) but am now convinced that wasn’t due to any bias but rather may be due to some self selection or exclusion earlier in the process. Are minority students pursuing fields of studies that would lend themselves to careers in intelligence analysts at lower rates than they should?
While this may sound like a lengthy, involved process we found that it actually took no more time (after developing our criteria) than the old system which yielded consistently unsatisfactory results.
One important final point…This was only the beginning of a long process to develop quality analysts. After hiring, the candidates went through an standardized orientation and a lengthy (if unofficial) on the job professional development/mentoring effort. None of that, however, would have born fruit had the right candidates not been selected.