Three ways not to hire ethical hackers
Most high paid technology workers have expensive, time consuming bachelor's degrees. For a few different reasons at the moment, employers continue to value such degrees and prospective programmers find it cheap enough to placate them.
Unfortunately for the traditional employer, however, most universities haven't gotten around to building educational programs for information security professionals. Malware reverse engineering, vulnerability research, red teaming, DevSecOps, and network security administration are all somewhat independent skills that each demand years of study. Yet, when the people to hire for these positions have degrees at all, they tend to have computer science degrees, and not specialized cybersecurity degrees. People without specialized domain experience thus have a hard time selecting for those who do, and so when businesses that aren't cyber need cyber (like most large companies nowadays), oftentimes they decide to erroneously select based on the following things:
1: Security certifications
The most common credential asked for on online listings for cybersecurity positions, besides degrees that don't include information security content as apart of their pedigree, are far and away private certifications. Security+,CISSP, OSCP, CEH; acronyms dominate the names of LinkedIN security researchers like they do doctors. Unfortunately for employers, one might see this and be mislead into considering this an endorsement of certifications from security workers, rather than a resigning submission to the status quo. It is very easy to list problems with these products individually. The CEH, Security+ and CISSP certs test people entirely through multiple choice, handwritten (or online) exams, and make no attempt to filter applicants based on any actual task they're asked to perform. They are about as effective at filtering applicants as a quiz on the internals of NodeJS is at filtering web developers. None of them make any attempt to test a security professional’s competency at demonstrating vulnerabilities in new products, which is the core distinguishing feature of good penetration testers, not knowledge.
CISSP in particular does require a minimum of five years experience, and experience is good, but the actual test itself ensures better literacy in cybersecurity philosophy more than your candidates' vulnerability audits. Otherwise, you are better off just requiring 5 years’ experience in the first place, and forgetting about the CISSP entirely. The OSCP on the other hand - rare for a certification - does to its credit test its users with a genuine practical exam: oftentimes by asking them to hack virtual machines created decades ago, on operating systems like Windows Vista and Ubuntu 12.04. You may learn from any one of these that the applicant is willing to go through the motions for professional success (and most hackers do in private bash these certifications), or perhaps that they are within a lower bound of competence, but otherwise these are not very strong signals. All of the above tests, by the way, also allow unlimited retakes, and in some cases let examinees answer repeat questions or undertake repeat tasks on subsequent exams. At their core, though, these certifications remain unhelpful not because the content is perpetually outdated, or because you can't test technical skills very well via scantrons, but because the organizations that provide them are not really incentivized to be effective.
The private companies that make these examinations are paid based on the number of people who attempt to take them. Certification companies' revenue is not, for example, tied to the amount of their takers who get job offers, their salaries, or their internal performance evaluations. They are in the business of making their tests look difficult and selective, which is different than making them actually useful. As far as I can tell, none of the standards bodies or for profit businesses that maintain these certs even perform the gesture of weighting their questions by pass/fail rate, let alone tracking the metrics above. I've earned both my OSCP and OSCE, and while I thought the courses were fun and the exams an interesting experience, I certainly haven't been asked by Offensive Security whether or not these certs got me a job. Why would they bother? Either way the thousands of dollars I've given them stay where they are.
2: Referrals
This technique isn't unique to computer security. Other fields also struggle with finding good talent through referrals, and, in general, asking your peers can provide a useful initial source of candidates when making hiring decisions. The difference with cybersecurity, though, is twofold.
First, it's far more often in this industry that the company hiring for a cybersecurity position or requesting an audit is not themselves a security company, or even an information technology company. For most professions, in evaluating a technical skill like the ability to monitor a network, people have a cone of uncertainty centered around their own level of expertise. If you are an average grade mathematician, you may understand who the best mathematicians are in your department, but you will have a harder time evaluating via interview who the best person is across a dozen different departments. If you are not a mathematician --- if you're a baker and you find yourself in need of mathematicians for some reason --- you will have a hard time discriminating even between the average mathematicians. Trying to evaluate people through referrals thus becomes a chicken and egg dilemma; John says to hire Lisa, but to appreciate his advice you'll need to first evaluate how competent John is, and you can't make that call unless you yourself know enough about what John does to hire Lisa in the first place.
I've found that this is a larger problem among IT companies looking for help with their security posture than it is among the bakers, because bakers usually understand that they can't effectively grade computer hackers. "Technology" companies on the other hand will often severely underestimate the skill gap between their field and my field, and figure they can handle themselves just fine until their first security scare.
The second problem with using referrals in this industry is that here there are entire networks of frauds. I don't mean frauds in the dishonest sense; I mean the much more dangerous problem of frauds who don't realize they are frauds.
One of the reasons I strongly recommend red team experience in blue teamers is because legions of "information security professionals" go to work every day with no real idea in their head of what a breach looks like. Chefs that cook bad food can be fired immediately, if at cost. But in cybersecurity, you can hold onto entire departments that for years have no clue what they're doing until the firm loses millions, because it’s so uncommon that (especially blue team) personnel are systematically evaluated. Entire moral panics are founded in this industry around unweaponized bugs like Spectre and Meltdown, because everyone is constantly engaged in a game to inflate each others' self-importance. And note for the wise: hiring based on experience is, in effect, hiring based off of their previous company's pseudo-referral.
3: The dodgy, in-house hacking gamut
Let's say for the moment that you do know web application exploitation. Perhaps your company is in the penetration testing or consulting business. How do you create a rigorous system for filtering applicants for that position, if you want to do better than gut feelings?
You wouldn't want to make the decision based simply off of their ability to talk to you about the field for a few minutes, or else you'd be better off using the CEH. You might start by building a series of timed, live challenges for the sub discipline you're working to test. By challenges, I mean presenting them with actual applications or networks that they haven’t seen before, with vulnerabilities they’re meant to find or exploit. You’re not looking to make CTFs here — ideally these should be based on actual bug bounties or bugs found in the wild, planted in as realistically fully fleshed apps as feasible. This way, after some optional preceding screens, you can actually make a practical assessment of the competence of the person you’re considering giving tens of thousands of dollars to protect or assess cybersecurity.
It’s also not enough to just have a series of vulnerable VMs that you show your applicants, even ones you have a consistent rubric for grading. If you want to build a good standardized test, which is what you’re trying to build, behind it needs to be a statistical model that explains how performance on these challenges relates to actual work performance. By that I mean you need a mathematical system for how you interpret the results, that doesn’t just devolve to you assigning point totals to each web app and allowing in anyone with a score of XX%. Your test needs to get more definition as you get data on it from incoming applicants. For instance, a good grading system that passing challenges that get consistently failed (indicative of strong ability) is a good sign, and decreases weight on challenges that don’t seem to correlate with performance on any others (indicative of irrelevance or being too niche). The best types of these models allow you to thus perform adaptive testing, where your applicants are matched with a new challenge at the median of their estimated ability level, allowing you to get to do a sort of “binary search” of their skill. Since these hacking gamuts are several hours of your applicants’ time at least, it greatly benefits you to be able to squeeze the most information out of each of these sometimes more-than-two-hour-long practical problems. Then, you need to engage in genuine followup, where you track candidate performance and make sure these challenges are actually measuring the thing you want them to measure, and rejecting the applicants they out to be rejecting.
And of course, over time, you’ll need to have both another system for sun-setting and updating challenges that start to become outdated (as they absolutely will), as well as one proctoring these exams so that your applicants can do them remotely without you physically standing over their shoulder. If you don’t have both of these things, your exam is going to end up irrelevant, or cheated, respectively.
If this sounds like a lot of time and expense, that’s because it is. Needless to say, speaking as a founder of an aptitude testing firm that does this for others, all of the companies we approach that actually maintain their own in-house challenges fail one or more of these big ticket items. In fact, after consulting a dozen or so companies about this (some of whom we respect greatly as hackers!), we can safely say we’ve never seen any serious attempt to mathematize the pass/fail system of these challenge courses at all. Far from it, sometimes we find that their challenges lack a rubric entirely, or else that the company has a hard time even figuring out whether or not they got attempted by their applicants in the first place. If you’ve done one of these before, there was probably nothing behind it besides a pentesting lead that ended up making a gut call. Which means measured ability of candidates tends to become another vague credential alongside education, years of experience, and certifications, rather than the main deciding factor it ought to be, given that it’s the most objective information you have when making the decision to hire someone. It takes a lot of work to make your work aptitude tests scientific, but it’s the only way you’ll be confident enough to see real value out of them.
If you don’t have time to make all of this, and don’t want to hire an aptitude testing company like Leonard, how are you supposed to make these sorts of hiring decisions? We’ve found two alternatives, neither of them ideal. The first is the traditional strategy of punting the problem down the road by forcing your hires to go through some sort of trial period, where they’re evaluated on-the-job for three months to a year. This has the obvious downside where if you’re wrong about a person based on their credentials and the interview, you’ll probably be wasting tens of thousands of dollars. The not-so-obvious downside is that this limits who you can hire to people you’re willing to risk on this highly costly live-fire assessment, which means you’re losing out on most of the underrated talent. In addition, it’s a strategy that can only really be practiced by firms that already have a way of somewhat impartially assessing the performance of their hires. Blue teams don’t hire this way, because only the teams lucky enough to encounter a breach can perform good nose to the grindstone evaluation of candidates.
The other is to limit your incoming hires to people referred to you by others you know are good (because you yourself are a great computer hacker), or those with very strong track records — highly ranked bug bounty hunters, former SpecterOps people, etc. The obvious issue here is that you’re rejecting the majority of good potential employees, and you’re biting the opportunity cost of not having positions filled in the short term. The other issue is that, once you run out of referrals, you’ll invariably be underwhelmed with the person hired this latter way. This is because the first person that comes across your desk is going to be the person who minimally meets all of these credentials, that the other firms neglected to hire because of some outstanding issue. There has to be some reason that this “ex-NSA” badass (read: regular government employee) decided to come to your company instead of everybody else. If you say you’ve found a star but only passably-credentialed hacker via an excellent internal assessment tool, or after a trial period working for you, that makes sense, because other firms probably wouldn’t have the information you have simply by looking at that employee’s LinkedIn profile. If you’re making that determination of excellence based on something freely available on the persons’ resume, then you have to ask yourself why they aren’t already employed somewhere else. There can be valid reasons, but hiring is normally a lot like real estate: overpriced houses stay on the market, and cheap houses get bought up quickly.
Otherwise, there’s really no other way to build a top-tier security consulting firm. Every growing penetration testing or appsec company is doing one of these three things, and usually a combination of them. Hiring in security is risky, but the companies that maintain and consistently improve a solid hiring pipeline are always the ones that end up succeeding in the long run.
