Hiring The CSO

Advancing a security mission with new leadership

Waiting for a CSO to join a company is not a good reason to delay the creation of a security program. Most security work can be done without leadership. Of course, leadership is still useful.

Be sure to set expectations here, as it’s a common mistake to think a CSO will relieve a company of all the tasks needed to be secure. You’re hiring something very different than a builder of an organization. They’ll be involved with the whole company in different ways.

Let’s talk about the many qualities of a leader who can drive the mission of security at a company.

I’ll call it a CSO — though this can be a VP, Director, Manager, etc.

TEAM PLAYER

The security talent and background of your candidate doesn’t matter if your fear of a breach allows you to hire a toxic individual.

Toxic security professional have a few ways of sneaking in.

First, a good FUD slinger may win themselves the position if you’re vulnerable to hiring someone “scary”. You’re not looking for a doomsayer, you’re looking for a castlebuilder.

Second, if you’ve lowered your overall standards due to a security breach, you’re opening yourself up to bad hiring. Try to maintain your normal standards around culture and team fit, and don’t excuse a specialist who deviates too far from your norms because you needed help in a rush.

Questions: Standard team fit questions around leadership style, management technique, company philosophy, their motivations to secure your company and motivations to pursue a security career.

CONFRONTS SKELETONS

A bad candidate will be itchy to execute on disparate projects before they’ve scoured the company for embarrassing debt. A candidate that probes for debt with sharp questions should be taken as a good sign. Upon joining, they’ll want to be briefed on every major incident that has happened so far, every close call, and will not make assumptions on what controls you may have implemented.

They should not start working on risks in the order of how they found them. During an interview, ask them to describe a program that finds issues, prioritizes mitigations, and eliminates risks while preventing them from coming back.

Don’t hire a candidate looking to repeat their previous solutions. Risks drive a roadmap, not previous experience and what comes easy to them.

Questions: What is your first week going to look like? What information you need to have access to, or know? What risks will you tackle first? How would you approach these “x” risks?

RECRUITS TALENT

A strong CSO will suggest candidates for partner teams that collaborate well with their mission, like an IT help desk or a network engineering team. If they only desire to fill up Security with talent, they may be empire building. A CSO’s role should be significantly easier if they support partner teams within the company to the point that they’d be willing to refer candidates to them, too.

“Security” is a concept, and not an organization. A CSO should help build other organizations as much as their own.

More importantly, a strong CSO can pitch candidates on why it’s important to protect the company’s mission. Once they’re totally sold on blockchains, streamlining car rentals, safer one-night-stands, or whatever it is your company is about… they should be willing to take bullets for the mission with a straight face.

Lastly, typical of any role, a CSO should be able to identify a weak recruiting organization and support them however they can. This goes for any leadership role, really, but engineer recruiting in security is extremely hard as the pool is very specialized and usually locked into their current roles.

Questions: What’s your plan for team building? What attracts you to our mission, and how would you explain our mission? What areas of our org are critical to your success? How do you partner and get along with those organizations?

DEALS WITH FRAGMENTATION

Usually the “secure way” to do something is very clear. A CSO’s job becomes muddled by scale, geographic disparity, acquisitions, or a million other complications.

Ask your candidates to describe a time when a mitigation was not as straightforward as applying the most reasonable best practice. Weak candidates can list off best practices very quickly and confidently, but a strong candidate can list the crazy hiccups common during deployment.

An example: Multifactor is a well accepted best practice to have. When Facebook launched internal multifactor authentication, DVORAK keyboards caused a support headache as they deployed across their engineering org, making a common best practice easier said than done.

A weak candidate with strictly consultant or compliance experience will only be familiar with advising “we must have multifactor”. They’ll come without this sort of deployment war story, maybe from inability to deploy at previous employers, or from inheriting a solution from previous leadership. A better CSO will have implementation experience, either hands on or from true involvement with their previous employers.

Questions: Can you describe how your previous workplaces complicated a typical security practice? What mitigations that should have been simple ended up being headaches? What risks do you feel are the hardest to mitigate properly, and for what reason? What areas do common solutions lack?

TAKES RISKS

A toxic security professional will continuously side with a “avoid the risk” position until the lights turn off. Security is actually a job tasked with taking risks, which too few like to admit, or believe. It’s why so many leaders dive bomb their security organizations into exclusion.

Here’s an example, with a personal story of mine:

In 2007(?) I told co-workers that an early design of Facebook Connect would increase the risk of phishing attacks due to its placement of our authentication UI on non-FB domains. Of course, it was a early draft and not up to par. I felt strongly that it was so far off that we should move on and ditch the whole idea altogether.

But, security was part of product development and iterations happened came through. I worked with the engineers on removing the big design flaws until I realized that this product would actually carry a significant defense against phishing. Increasing legitimate engagement meant that we could more easily identify a phishing attack in progress.

If the product succeeded — our users would become more secure. A risk worth taking.

To explain further, banks may only see authentications from their users on a weekly or monthly basis. They have little information to base detection of a suspicious logins on. Any endeavor to increase logins to Facebook would greater polarize suspicious logins into easier detection methods, as we’d have more “good” to compare against “bad”.

The back end systems eventually supporting this theory became a part of a system that prevents, to this day, an exorbitant amount of account takeovers, spam, and other badness. Over time, classic phishing became statistically insignificant and replaced by more determined (smaller) set of malware based attacks on hosts and in browsers.

I ask myself frequently if I could have been the guy that doomed Facebook’s growth if I was just a little bit more impatient, or if I was in a CSO role with an archaic veto power. What a horrible mistake that could have been.

Facebook Connect ended up becoming a strategic growth driver during a critical period of competition and I’m glad I didn’t fuck that up.

Questions: In what cases have you vetoed a project? What causes you to vehemently side against another team? What sort of conflicts existed between your team and others? To what extent should an insecure product, system, etc, be supported by security?

AN INCIDENT RESPONSE VETERAN

Ask your candidate to share incident war stories. Be careful though, someone who wildly violates their trust is a bad sign. On the other hand, someone who overuses the NDA excuse to shield you from their experiences is also a bad sign, they are usually protecting their own inexperience. Any clueful candidate can easily speak around attribution without harm to anyone.

If they claim they’ve never suffered any level of breach, run far away from that candidate. Especially if they claim it’s due to their own amazing defenses.

Within a larger company, a strong candidate is someone who can manage the operational issues of varying incidents at any time. You may even have security teams that don’t report to the CSO that you’ll need to play well with.

For the smaller company, you need someone who can help calm and manage companywide panic. It will also help to have a diverse network of contractors for rapid response capability (forensics, reverse engineering, etc). They’ll need to be the clear leader that can help communicate issues to customers, the blog, etc.

Questions: How do you support a team mid-response? How have you prepared a team for a breach, emotionally? Have you ever run a Red Team? What methods of intrusion have you experienced? Describe to me what steps you’d take if [some system] was discovered to be compromised.

FEAR COMPLIANCE

Some candidates will use compliance frameworks as their guide to a security program, especially from government or finance. You’ll find that candidates who have actually responded to breaches (and care to prevent them) avoid compliance inspired security programs.

You will want someone who is driven towards security because of the risk of intrusion, not someone who has practiced security to achieve a certain level of compliance for the business. If the compliance workload in your company is significant, then you’ll want someone (maybe not in the leadership role) who can track this. If compliance is a business driver and have nothing to do with actual security, so be sure to distinguish this when asking about your candidates experience.

Question: When has compliance gotten in the way of reducing risk? How can compliance help us reduce risk? How have you dealt with auditors in disagreements about mitigations and risks?

NO CHARLATANS

There are plenty of CSO’s who are continuously on the road enjoying a guru lifestyle. The security industry has a comprehensive conference circuit that pays for speakers. Having a periodic speaker engagement on the resume is great, but there are plenty who live out of the airport and give the same powerpoint to rooms full of security managers on boondoggle.

Try to press on a candidates involvement in the greater security community and have them detail their work products as a result. If they’re constantly vacationing on the back of the industry or community, it’s a strong bad sign.

This type of traveling is more plausible if there are enormous trust perception issues with the company, if recruiting is a priority (and their efforts are paying off) or if the role has some sort of lobbying angle to it. All of these benefits trade against their active leadership potential. If they aren’t present, they aren’t leading.

Questions: What are your speaking goals? What are your goals for personal visibility? How will that impact your presence with our employees? What conferences do you visit and what opportunity is there?

CREATES TECHNOLOGY

Security isn’t a product you can buy. A candidate should have a trail of homegrown mitigations they can speak about. They’ll be able to nurture an environment where solutions are developed in lieu of a purchased solution.

If a candidate can direct a high quality engineering organization, you have struck gold. Many good examples of homegrown security tools appear on Github, from AOL, Facebook, Etsy, Netflix, etc. These tools are symptomatic of a strong engineering team behind them.

Questions: What have you built to reduce risk when no product was available to help you? Have you ever managed an engineering organization? What is your engineering background like, as far as building defenses or software?

Conclusion

Find a CSO who has dealt with breaches, can manage a technical team, builds independence from proprietary products, and follows risks instead of checklists.

Most good CSO’s touch into theses areas, but rarely hit all of them. That’s just fine. Besides being a hiring guide for a company looking for a CSO, I’m hoping this will help those looking to enter security leadership as well, because we all need better leaders instead of more of them.


@magoo

I’m a security guy, former Facebook, Coinbase, and currently an advisor and consultant for a handful of startups. Incident Response and security team building is generally my thing, but I’m mostly all over the place.