A guide on how to successfully build, structure, hire, and retain an Application Security (AppSec) team in a modern business.
By: James Chiappetta
Disclaimer: The opinions stated here are my own, not necessarily those of my employer.
The modern business is one that is building technology with applications in cloud environments. The need to ensure that these applications are built with security from the start is now top of mind. Organizations are unfortunately faced with some difficult challenges in doing this. It’s time to look at how an organization structures itself with Application Security (AppSec) as a key part and what it takes to build a durable AppSec team that will last forever.
Note: If you are looking for guidance on security services then please see my other posts. I’d recommend beginning with How to Kick Start your AppSec Program with Ease.
First Things First
It’s now 2020 and Covid-19 has forced a lot of companies that were not traditionally software companies into the age of digital product delivery. One of the main ways the industry has done this well is with DevSecOps. The primary benefit of this style of operation is the coupling of cross cutting product delivery teams (Product, Dev, Security, and Infrastructure) and their ability to bring high quality products to the market faster. This is Conway’s Law in action. Sounds good, right? Sure, but how does the AppSec team fit in and how can they be set up for success for the long term? To answer this, we need to cover a few high level areas:
- Fitting AppSec into the organization and the role of the security leader.
- Defining the vision to attract the right talent and getting the right people hired.
- Setting up the team for success.
- Building a brand and addressing the challenges of long term talent development.
Before we get started, it’s important to note that this post is meant to reinforce familiar concepts in team building, while highlighting the nuances with setting up a highly specialized security function. Ok, now let’s get to it!
Optimizing the organizational design for AppSec — A 2020 survey of CISOs by Hitch Partners found that the CISO reporting structure varies greatly and that: “There seems to be no clear answer as to the best home for the modern CISO”. Based on my experience, I agree and will come back to this topic in more detail in the future. What’s important here is that:
- A proper security program should have a CISO.
- The AppSec team needs a clear line of sight to support all of the critical business functions in order to be successful.
Here is what that could look like:
Defining a Vision
The Vision Statement — If you want to attract the right talent, be transparent on the vision of what AppSec will mean for the business long term. Be honest about where things stand today but really focus on the vision of the desired state. This is going to be a journey!
“Men wanted for hazardous journey. Low wages, bitter cold, long hours of complete darkness. Safe return doubtful. Honour and recognition in event of success.” — Ernest Shackleton
Here is an example of a AppSec vision statement:
We are looking for collaborative, curious, and passionate people to join our Application Security team. The team will need to build enduring processes with innovative technology. We seek to improve the safety of customer data, provide innovative, yet seamless security services to the company, contribute to the community, and create long lasting relationships.
Pro Tip: Put your vision statement front and center on your job postings.
Building a talent pipeline — This is not easy, especially if you are at a smaller company without a strong household name. This is where your vision statement will be key. If your organization doesn’t have a robust internal recruiting team then you may want to leverage an established external hiring agency. In the Cybersecurity and DevOps space, you may need both internal and external recruiting for highly specialized roles. Getting momentum in the talent market is hard when you are starting from zero. You will pay a premium getting help but it’s worth it in the long run as the cybersecurity community is not a big one. It will help in the long run because word of mouth can be a key difference in hiring top talent.
Diversity — The most successful teams I have ever built and been a part of have all been diverse. This means a lot of things but most importantly, different genders, ethnicities, cultures, backgrounds, and technical skill sets. Many different perspectives and ways of thinking about the same problems will always result in stronger outcomes. This is precisely why I have people peer review my content before I publish it on Medium!
“You may be missing out on potentially valuable ways that employees can contribute to their organizations“. — Hitch Partners
Interview process — Every company runs interviews differently. In general, an interview process that balances technical skills and culture fit works best. Both areas are vital and need eyes from the existing staff and key stakeholders such as existing security team members, dev or infra engineers, and key leaders. This will ensure the right candidates get selected for the team. I find the best candidates demonstrate how they have focused on building strong relationships, earning trust, and continuous learning; while nailing the fundamentals of technology and security.
Setting Up for Success
Mission & values — Most companies have a mission and core set of values. Having a derivative of those for your AppSec team will help create cohesion between the top level company values and the team. Establish a mission statement and values together as a team. This will provide the invisible fabric that connects the team every single day.
Here is an example:
Mission: Enable the business with seamless, yet innovative, security services that keeps applications and data safe.
- Trust — Earn it, keep it, and build upon it
- Empathy — We are all in this together
- Transparency — Think aloud and communicate with confidence
- Iteration — Do the simple things first and build from there
- Innovation — Dream big and think creatively
Strategy & goals — With the vision in mind, it’s time to break into strategy mode and figure out the priorities and sequencing that makes sense for your team. Pair goals up with the people on the team that will benefit the most. Also pair people up for mentorship and collaboration. This will result in even better outcomes. Don’t forget, a highly successful AppSec Program is one that is Agile and built with high levels of trust with its stakeholders.
Roles & responsibilities — In a DevSecOps world it is in your best interest to work to define who does what, both internally on the AppSec team and with your stakeholders. It’s important to get agreement from your partners on where responsibilities begin and end. This helps avoid confusion and annoyance.
Team capabilities & tools — Define the services you want to offer the business and how they fit in. Then work as a team to define the tool set needed to get the work done. This will help you create a service catalog and get the budget.
Planning for the Long Term
Team fulfillment and growth — It’s crucial to keep your AppSec Engineers’ career growth and the overall team’s growth top of mind. The following list is a set of ideas that can be set up in order to make the interactions and experiences on the team be as fulfilling as possible:
- Set up regular non-working lunches.
- Do a team hackathon.
- Start regular team coding/development day (one a month).
- Pair up and go to security classes or training (e.g. SANS or OSCP/OSCE).
- Go to security conferences together (e.g. Blackhat or Shmoocon).
- Promote team members to a Team Lead role where they can own a domain or service area on the team (e.g. pentesting or design reviews).
This will help individuals on the team share knowledge and mentor each other while contributing to the greater good of the team. These activities are crucial for cohesion as it flows through the Tuckman model of development (forming, storming, norming, and performing).
Avoid the pitfalls of the deficiencies in the cybersecurity talent pool — There is a growing shortage of people getting into the Cybersecurity space. Here are a few ways you can keep multiple pipelines healthy and alive:
- Start an Internship program.
- Create career paths from other teams in the organization such as IT Help desks or other support/ops teams.
- Start an Engineering Rotation Program for others in Technology to rotate on the AppSec team for a quarter.
- Get buy-in for training and education for the team and anyone else in the company.
Building a Brand
Creating awareness for AppSec — You have a team, a mission, a strategy, and a set of goals; congrats! Get the word out about the team, what you have planned, and how others can engage with the team. There are many ways you can start building a strong reputation and brand. Here are a few:
- Hold lunch and learn sessions.
- Create an internal wiki to host team content (e.g. secure coding checklists or secure development guidelines).
- Start holding regular office hours.
- Create a Security Champions group (more on this in a future post).
Importance of dealing with cultural conflicts and fast — It’s perfectly normal to have differences and conflicts pop up in every which direction. It may be between two people on the AppSec team or with stakeholders. These are addressable and it’s so important that these differences are addressed expeditiously. If you let divide build, trust will degrade, and you will end up in the The AppSec Doom Loop. This will cost the company and lead to less than ideal work experiences.
Pro Tip: Dealing with internal conflict, under performance, and highly jaded employees is equally as important to the health of the team. Work with your internal HR Business Partner on how to best address performance and any toxic effects of those who may be negatively impacting the team.
Application Security Team Development Cycle
Pulling it all together creates a team development cycle which looks this —
- Ensure accountability and continuity between AppSec and the teams they support. Clear and well understood reporting lines are key in the DevSecOps model. Together is better!
- Define the vision statement for your job postings and team. You need to get the right people hired and make sure diversity is built in from the start for strong outcomes.
- Work as a team to define the mission, values, goals, roles, and capabilities together. This is the AppSec Team Development Cycle.
- Start building talent pipelines for future team growth and keep the talent you have fulfilled.
- Make sure those in the business know who the AppSec team is and what they do.
- Deal with conflicts fast as this can erode the team’s reputation and trust.
Words of Wisdom
Everyone will have a different formula for what right looks like. Much like everything in life, there are balances and compromises that need to be made all over the place. Keep your mind open and embrace the inevitable flow of change that will help nudge your AppSec team in a positive direction.
Do not let past org design models continue to dictate how things need or should be. Your AppSec team will no doubt thrive with trust, empathy, and enablement being at the forefront of what they represent. At all costs, avoid The AppSec Doom Loop and deal with cultural conflicts fast so trust doesn’t erode.
Want to know how to properly handle product usability with security? Then check out my next post: How AppSec Can Help Balance Product Usability With Security.
Contributions and Thanks
A special thanks to you, the reader. While this post is on the longer side, I hope you benefited from it in some way. I want everyone to be successful at this. While these posts aren’t a silver bullet, I hope they get you started.
Please do follow my page if you enjoy this and my other posts. More to come!