Building an Application Security Team
[no description provided]
The Application Security Engineer role is in demand nowadays. Many job offers are available, but actual candidates are scarce. Why is that? It's not an easy task as the level of skills needed is both to be broad and specialized at the same time. Most of the offers are about one person, one unicorn that does all those wonderful things to ensure that the organization is making secure software.
Looking at where we are coming from
For the sake of simplicity, let's say that the application security industry is traditionally split between two main types of offering: those who are selling products and those who are selling pen test services. In many occasions both are from the same vendor, but from different teams internally. Let’s break down how they are judged on their success to have an idea how it managed to evolve.
For the products approach, we must concede that the actual work is executed by a computer program. It requires a certain level of intelligence from software, but at least we can have a certain amount of codified knowledge that can scale and work 24/7 to ensure larger coverage. Ironically enough, one of the focus of that machine seems to be somewhat the same as many talented individuals want for themselves: to be right. Success is defined when the tool finds the correct issues, while not annoying people too much with false positive.
For pen test services, work can be over simplified into running tools as well, but the value is also when the amount of false positives is quite low. Areal human that does a proof of concept can ensure that, plus it makes a great thing to show off. Also, a lot of creativity and adapted deep technical knowledge need to be applied dynamically, a thing that most tools don’t achieve at all. The findings will then be amplified and the actual coverage will not matter as much. A pen test can be deemed successful if it finds a few critical issues without having a 100% coverage. To be fair, many pen test services will try to achieve full coverage. They will also work in a black box model, making it a difficult endeavor. The evolution of pen testing seems to be heading toward acting as a Red Team; a team that focuses on mimicking what attackers does. In this case, all we need for success is one or a few open doors in the right place at the right time.
Both tools and pen testing service are useful and should be the first thing you try when you want to address the "security problem". They also both always provides recommendations on how to fix. Those recommendations tend to be simple and tied only to issues found. They might not be on point with identifying the root cause of the problem and often come too late in the development process. Since security tools and pen testers know best about security, and that knowledge is humongous to grasp, not much is left for the knowledge of building secure software. That is part of defining what is success in the software industry and, while they are here to help, the security industry has it as a byproduct of its own success.
It makes sense that an industry who wants solve a problem, tries to solve it like it has always done before: by using exceptional talents doing over-specialization. Unfortunately, the efforts required to fix are not from the realm of security engineering, but from the broader realm of software engineering. This is yet another large body of knowledge where only a fraction overlap with security.
A partial view of the industry
So before going on with what I personally think, I want to explore how the industry defines the role and tasks of an application security engineer.
Fair warning, what I've done here is a totally biased research™. The bias I can see is the following: I’ve collected all the job offers that one individual has received in the past two years while identifying as Sr. Application Security Engineer on a popular social media platform that is the hunting ground for recruiters. It is not representative of the industry at wide at all, but rather a view of what someone would attract in a specific geographical region. What I like about that bias, is that it expresses a consensus of people looking for the same thing, so really pin pointed at what I want to explicit in this post.
Let’s look at what trend we can extract from a total number of 57 job offers.
- 35% (20) have the title “Application Security Engineer”
- 11% (6) have the title “Application Security Architect”
- 4% (2) includes “Product Security” in the title instead of "Application Security"
- 7% (4) are about management or the title “Application Security Manager”
- 21% (12) have senior in the title
- 11% (6) have the title “Security Engineer”
- 2% (1) have the title “Software Engineer”
- 19% (11) talk about an AppSec program, SDL or SDLC
- 5% (3) talk about pen testing
- 18% (10) talk about code review
- 9% (5) talk about threat modeling
- 12% (7) perform development/building tasks
- 26% (15) are far from the regional position
- 35% (20) didn't provide details about the role
- 3% (3) mentioned an application security team
A few years ago, I was barely seeing the “Application Security Engineer” title, but now it has become more predominant. That change had probably the effect to put the role in a high demand situation as few people has it in their background history.
For more complete offerings that include many activities such as threat modeling and code review, the offer is also low, as it is hard to find a single individual that can and want to tackle everything on the table. The fact that we rely on one role also make it hard to institutionalize, and that doesn’t help at all at the resource management problem we now face with that offer-to-demand ratio.
The core of this current blog post is about building a team, and that’s for me a solution for easier recruiting and better security impact overall.
Enough with contextualization, let’s go on the subject!
So here we go, here's the list of 11 roles I can think of, with sample responsibilities and skills requirements. A role in this case is not a job title but rather a label that is easy to use and refer to.
In order for the list to make sense, you must have some prior knowledge in application security programs that define diverse activities during the software development life cycle (SDLC). OpenSAMM and BSIMM are good ressources to help you enumerate and understand those practices.
| Roles | Responsibilities | Requirements |
|---|---|---|
The Hacker |
Break the applications Do manual and dynamic security testing Provide insight on threats |
Participated in bug bounties in the past Knows the threats related to your target applications |
The Secure Developer |
Code Review Guideline for fixing Contribute to code changes if needed |
Deep knowledge of tech stack you are using Knows the pitfalls in programming languages and frameworks |
The Organizer |
Keep in touch with development & other security teams Manage schedule and planning |
Strong communication skills Project management experience |
The Architect |
Lead Threat Modeling Steer secure architecture changes in the entreprise |
Experiences with systems related to the target applications Good with creation of diagrams |
The Automation Engineer |
Integrate tools in CI/CD Work on automation of the team's tasks |
Proficiency in scripting language Experience in DevOps |
The Vulnerability Researcher |
Digg in the details of some systems Elaborate proof of concepts when needed Find pin pointed vulnerabilities |
Deep knowledge of operating system internals Knowledge of exploitation techniques |
The Apprentice |
Anything to assist other roles Help with team motivation and staying current |
Is highly motivated by the objectives of the team Quick learner |
The Cheerleader
|
Provide positive focus for the team Ensure team motivation and well-being Detail the service offering of the team to customers |
High charisma Marketing background Good at karaoke |
The Teacher |
Prepare and execute training Identify what needs attention for education |
Great communication skills Good charisma Knowledgeable in security |
The Documentation Artist |
Helper for training and documentation Design and polish documents |
Some Photoshop skills Some Office suite skills Reads a lot about security and/or programming |
The AppSec Lead |
Builds the Application Security Program Can do most of the other tasks by the team |
Proven vision and leadership skills Knowledge of SDLC Knowledge of Security |
Of course, those are stereotypical as a real person can take on many roles and responsibilities can be swapped between team members. Too many roles played at the same time (over a quarter or sprint, depending on your velocity) will dilute the competence of the person. Having one person do everything might make sense when the target size of the customer pool is small (<10) but doesn’t cut it on large teams.
To be successful, I'm arguing that only some combinations are to be considered optimal, that is called the Most Effective Tactic Available (META). That META will surely change with the evolution of the industry and the actual application security practice in the target enterprise.
For example, you probably need way more junior level technical people first if the company has never taking care of software security before. They will be great at finding low hanging fruits. They can even pin point emergent security issues, that will seem simple to them, in a complex system by having an external point of view. The good thing about this, is that over time, those people will evolve into more senior roles. The team will then have less junior roles as the problems left will be harder to solve.
I’m proposing that those META should be relative to the maturity level of the security program in the organization. The exact number of individuals needed can be calculated from the number of developers†. Every entreprise will be different, but here's some examples that are realistic for a target application security team of about 4 to 6 people:
- The organization is new to Security; 3 Hackers, 2 Secure Developers, 1 Organizer (0 need for advanced roles such as Architect or Vulnerability Researcher)
- The organization already is composed of network security people; 2 Hackers, 2 Secure Developer (the Organizer will be part of what is already in place)
- The organization already have at least 2 well established Hackers (known as "pen testers") but none of the following; 2 Secure Developers, 2 Organizers
- The organization wants to focus on automation; 1 Hackers, 1 Secure Developer, 2 Automation Engineers
The reason you don't see an AppSec lead in the small proposed teams above is because that role can be seen as your wild card that can be used to replace single or even multiple roles. For example, you can probably exchange 1 Hacker and 1 Secure Developer for 1 AppSec Engineer for a good trade-off. We often see teams composed uniquely of those wild cards, but in this case the problem will be to have it grow to more than a couple members. If you want growth and have the enterprise establish a strong team over time, then the diverse META based approach will get you there.
Getting the right META for a particular group of customers is a difficult task. It can require a lot of observation of the current workplace reality at the enterprise and target group level. It is wrong to say that one META fits all, but you can sure find yours based on objectives and current security practices already in place.
There’s also a few good team tactics I can think of that can push the outcome of certain tasks to the next level. The synergy that you can get from team work will make the effective results better than the sum of its parts. Here's a few examples:
- The Hacker and the Vulnerability Researcher provide the Architect with threat intelligence
- Then the Architect has access to more knowledge regarding threat actors and their capabilities
- The Secure Developer works with the Hacker and Vulnerability Researcher
- Then the Secure Developer can implement more solid security controls by having them based on vulnerability knowledge and tested
- The Hacker works on finding vulnerability in a horizontal fashion (multiple applications) while the Vulnerability Researcher works vertically (only one application)
- The combinaison of the two will ensure better coverage as one dimension is often not enough
- The Organizer and the Cheerleader explains to less technically savvy people the impact of vulnerabilities and gain of fixing
- The fixing rate ratio will be higher for findings by the Hacker and proposed fixes by the Secure Developer
- The Architect validate its security decisions with the Vulnerability Researcher, the Hacker and the Secure Developer
- This makes the decisions stronger and introduce peer review within the team for more quality
- The Automation Engineer participate in finding issues with the Hacker, the Secure Developer or the Vulnerability Researcher
- This will make him better at creating or adapting rules for the tools
- The Teacher and the Documentation Artist work with everyone to gather needs based on security findings and best practices
- This ensure a consensus between the team in the vision and message communicated to software engineering teams
- The Cheerleader ensure that everyone else treats the customers with respect and good energy
- This minimize the impact of the "fix your sh*t" that we see so often by Hackers that are lacking empathy
- The AppSec lead utilize any other team member to empower its strengths and complement its weaknesses, while also covering the various responsibilities and tasks for an unfilled role
- This will ensure that no AppSec activities are left untouched as their usage can be beneficial even if done to a lesser degree than someone dedicated
Conclusion
It is hard if not realistic to gather a full team that fits exactly what you need. Most of the successful combinations will evolve around roles resembling those presented here. Asking all that knowledge, workload and versatility from one person will almost end up in making concession or burning out the candidate. It will inevitably make the growth of the application security team past a certain size very difficult if not impossible.
Bottom line, to achieve a great implementation of an application security strategy, diversity and team work is a must.
† BSIMM 8, a quantitative study of the software security industry, state that the average percentage ratio of software security people to developers is 1.60% with a median of 0.88%. Note that I think that this should be a minimum as proper staffing is a hard requirement for great success. The average size of a Software Security Group (or AppSec team) observed in BSIMM is 11.6, a pure coincidence related to my proposal of 11 roles.