Well. Errrr… Some do… Most don’t though! Without developers, penetration testers would not only be out of a job, but also miss out on a plethora of terrible, amazing, and hilarious stories to post all over Twitter and Reddit. No… penetration testers can’t have that…
Fine. Then why do developers THINK penetration testers hate them?
It’s complicated. Usually, it comes down to how developers seem to interact with penetration testers. Penetration tests can use up a developer’s time to assist with testing. Once a test is done, findings are generated. Findings take time away from feature development, the main metric used for performance reviews. Findings can also be used to make judgments on the quality of the code-base, the holistic application, or even the developers themselves. Now, Security should be considered a feature, not an afterthought. Developing value judgments of people based on test findings is also not constructive. Both true, and yet these situations still happen. If not altered, these realities tend to make developers defensive about opening their app up to penetration testers.
Developers often approach penetration testers with a defensive mindset. This breeds a whole host of defensive questions, assuming penetration testers are malicious and callous before the conversation even begins. A good penetration tester shouldn’t be out to get a development team though, so I’ve compiled a list of the top ten questions that developers seem to have for penetration testers.
Why is the penetration testing team testing my app? We didn’t ask to be tested?
There are many reasons that a penetration test might begin on your application.
- Maybe someone in your leadership chain wants it?
- Maybe someone in your security org wants it?
- Maybe an internal auditor wants it?
- Maybe an external auditor wants it?
- Maybe the penetration testers themselves determined that your app was a risk and thought it should be tested.
In the end, a penetration testing team wants to use their skills in identifying vulnerabilities to draw attention to issues needing fixing. They want to make the company more secure and someone has determined that your application should be prioritized. Why it’s prioritized comes down to the data that flows through the applications, who use the applications, and exceptions that surround the security of the application. None of that has anything to do with anything the developers did or didn’t do.
Why is the penetration team saying my code is crap?
They’re not… Probably…
A penetration tester that finds a vulnerability stemming from your code is saying the application has a vulnerability and nothing more. Any good penetration testing team wants to fix vulnerabilities by working with developers and will never blame a team or a team member for the findings in their app. Everyone makes mistakes and applications are complicated products. Beyond that, a pentester’s job is to find issues in an application. It’s valuable to design an application that doesn’t have vulnerabilities but pentesters specialize in finding vulnerabilities; they’re bound to find things a developer might miss. If a penteser doesn’t find vulnerabilities because you found them all before they tested the application, perhaps you should be on the penetration testing team instead? That aside, penetration testing is a service like spell-check. Their job is to protect you as a developer, your team, and your company from releasing insecure code into the wild.
If a penetration tester ever does never shame you for vulnerabilities, kick them in the shin a couple of times for good measure.
Why is the penetration testing team giving me this finding? They know we can’t fix it…
Knowing a vulnerability can’t be reasonably fixed by the development team in the required remediation window doesn’t mean pentesters shouldn’t report the vulnerability anyway. Security organizations and businesses of any size should have the ability to record both vulnerabilities and the risks of those vulnerabilities. The function of any penetration testing, application security, vulnerability management, or red team is to identify risks and it’s severities. It’s then the job of business and security management to weigh the risks and costs against the needs of the business. Knowing a vulnerability’s associated risks allows companies to account from them, resulting in informed decisions about their products and services.
Why doesn’t the penetration testing team help me with remediation?
Some penetration testing teams do assist with remediation. However, there are many ways to break up the overall responsibilities of a security team, remediation can actually be handled by other security teams as a way to allow specialization. An Information Security Officer can be both your security contact as well as your main go-to for security remediation assistance. This would allow a penetration testing team to be smaller or do more testing work since remediation can be time-consuming. To counter, a penetration testing team that finds the vulnerability could be more equipped to offer remediation advice they’re the vulnerability expert. Remember that a penetration testing team has their own job responsibilities and if remediation work isn’t part of them than any remediation assistance they offer takes time away from testing and securing other applications.
Nevertheless, the team best able to remediate the code will always be the developers themselves.
Why does the penetration testing team bother reporting such minor security issues? Don’t they know it takes time from doing more important work?
First, let's add context by stating this question usually references low, and note level findings. These findings are the lower severity issues and thus represent the lowest severity level findings that penetration testers care about. The classic example of this comes from server version disclosures. A server version disclosure is when an application discloses the version of the dependencies used or even the version of the application itself. This can lead to the identification of a vulnerable server version that the application is running but then that would be a different finding, “Vulnerable Component Version”. So why is “Server Version Disclosure” a separate finding? There are at least two main reasons. One, the disclosure of a server version provides assurance of a version, meaning an attacker doesn’t need to try attacks or exploits designed to determine what the version is. An attacker is less likely to be detected by monitoring systems if they exploit the correct type of attack once rather than loudly, numerous times. Two, knowing a version can save time by having to brute force guess the version in some way.
There is value in reporting low and note severity issues since findings can come together to create an attack path with a severity greater than the sum of its parts. Beyond that, it’s a pentester’s responsibility to the company to note the entire threat landscape, not just the ones that are the most severe. If the development team truly does feel like the finding is either too difficult to fix or not worth fixing, they can always accept the risk of the finding being exploited. As long as the team comes to handles a risk, it’s up to the business to desire how to proceed.
Why was this employee/person testing our application?
First and foremost, the identifier of the finding in question has no bearing on the fact that the finding is there in the first place. It’s common to have developer teams ask why the identifier was attacking their application in the first place rather than focusing on the issue itself. Understandable. To know that your application was attacked by someone is scary but please understand that asking for their identity is not conducive to the matter at hand.
Both Bug Bounty and Responsible Disclosure programs are blessings to a busy penetration testing team trying to frantically test all the things. They allow more resources to come in to test where there simply are not enough testers. Both programs employ at will testers. It’s in a penetration testing team's best interest to handle their contributions with care. As such, A penetration testing team should never disclose the identity of the identifier to a developer. If we want to make a company secure, siccing a disgruntled dev team on a good samaritan either internally or externally is a good way to ensure that the identifier will never disclose findings to the penetration testing team again.
As to why the person may have identified the vulnerability, there could be numerous contextual reasons. Maybe they’re a normal user and just had both the security knowledge and sense of responsibility to report the issue to whom it may concern. Maybe they’re a developer with security knowledge and a little curiosity that identified the issue during their poking around. Maybe it’s a tester in a bug bounty program and they identified an issue in your app to get a little pocket change. Regardless, WHY they are testing your application is less your concern and more of the security team’s concern. All a developer should worry about is working with us and the security team to fix the issue before it’s exploited. Don’t want a development team going after the desperately needed help.
Why doesn’t the penetration testing team set up a meeting? I don’t understand the findings in the report…
Maybe penetration testers will always set up summation meetings at your company. If so, this is certainly not the case everywhere. A lot of penetration testers will generate reports as this tends to be not only a good method of gathering all the needed information into one place but also as a stable record of what was identified, what were the conditions, and what occurred during testing. That said, meetings to understand the contents of the report are perfectly valid. They can highlight areas of a report that need more rework to better explain the topic in a language developers understand. It offers opportunities for a developer to give more context to findings allowing serveries to be altered or findings could even disappear entirely. Given the context and situational awareness, meetings can allow testers an opportunity to provide focused remediation and impact guidance for teams that proves invaluable for remediation and risk analysis efforts.
Where developer meeting requests break down is when said developer requests a meeting but fails to read the report. This happens a lot in busy companies and it’s an inefficient use of a pentester’s time. What's worse is when the people that need to be in the meeting aren’t in the setup meeting. They then set up another meeting where the “needed people” are in the new meeting, only to discover that they actually need another group of “needed people” that aren’t in THIS meeting. No matter the meeting, No one will have read the report prior because why would they when you can just read them the report? Reading is hard…
Seriously though, reading reports can be time-consuming and not everyone has the time during the day to read before every meeting. That’s perfectly understandable. However, please remember that your penetration tester is a busy bee and their time is as precious as yours. They might be doing other tests, handing retest requests, scoping future engagements, or plenty of other actives depending on the team responsibilities. Often a company’s penetration testing team is smaller than the needed number to sufficiently cover the developers, teams, and applications that the company supports. That’s not to say that developers aren’t worked as hard. A broken application during peak utilization can feel like Armageddon and that weekend hotfix can destroy sanity faster than the monsters of Cosmic Horror. Simply consider that pentesters often deal with teams that may not be as responsible as yours and simply coming to a meeting prepared with questions based on the report really encourages a pentester to help you when they can. After all, a pentester might have spent way too long on a report to be healthy and reading it really goes a long way towards appreciating their effort.
PLEASE HELP US!!! THE VAST MAJORITY OF REPORT GENERATION TOOLS ARE PRETTY TRASH… SAVE US FROM ETERNAL TORMENT…
Do we have to do this testing NOW? Our app is switching to a new account in 3 months. Can this testing wait till then?
Any more questions?
Really, applications change all the time. If a penetration testing team waited for all the changes in an application to be finished, no application would ever be tested. Besides, it can be a good thing to test an application when it is undergoing massive change. While it can be hard to keep the application stable, if a finding is identified, it can be easier to incorporate the remediation into your development schedule during massive change. Any remediations that require architectural changes are going to be easier to incorporate.
But you’re right. If your application is undergoing massive change right now, perhaps test now and test the application again when done? Food for thought.
I don’t think this finding is a high, can we lower it to a medium?
Push back is not a bad thing. Developers often know and understand context missed by a penetration tester during testing and it might bring up a point that actually mitigates the issue. Penetration testers shouldn’t stick to their guns without reason and if the context provided by a developer holds, then reassess the severity with that context in mind. Ego should be left at the door and Logic should reign.
However, know that asking for the severity to be lowered for a finding implies that a developer knows what that entails. Lowering the rating usually gives timeline extensions, giving a team more time to remediate the issue before the deadline. Lowering the rating usually means lower attention on the issue, meaning fewer people, and fewer IMPORTANT people will be pressuring them for remediation. Lowering the rating also implies the issue is less severe when it comes time for a team’s managers to do performance reviews.
For clarity, let's state it again. Neither findings nor severities should be used by managers to drive performance reviews. It promotes a toxic relationship with penetration testers and penetration tests that’s harmful to the overall security of the company.
That said, two issues arise around asking to lower severities. First, developers will often ask about lowering the severity to both put penetration testers on the defense as well as make that the topic of conversation rather than discussing remediation plans. Remediation is the goal and continuously arguing the severity of a finding not only takes time away from explaining how to remediate the finding but also doesn’t change the fact that it’s STILL a finding. Second, developers will often argue hypothetical likelihoods of attack and ease of exploitation because accepting a reduced likelihood or ease reduces the severity and thus the burden on the developers. If the only goal of talking hypothetical likelihoods and mitigations is to lower the severity of the issue and not to provide true context, then a penetration tester shouldn’t lower the severities.
Penetration testers are hired by the company, not the developer. They owe it to the company to enforce those deadlines and requirements when the issue warrants it. Similarly, they also owe it to the company to not distract developers from feature development if developers do provide valid mitigating context.
This finding doesn’t belong to my team nor my app, Why are they saying it’s mine?
Penetration testers are not omniscient. The delineation between the functions various teams are responsible for may not be clear cut and it’s very helpful to point a penetration testing team in the right direction as needed.
Note that in a lot of cases, teams will say their not responsible because they don’t want to be stuck with the finding and the remediation work. Here is a sampling of the top 5 unhelpful “that’s not our problem” responses
- “No that's not our finding cause it’s not in our app.” ( 2 days before due)
- “That's not our finding cause that's in a third party dependency” ( You use it in our app though…)
- “No, that’s the responsibility of the DevOps/Application team, not us.” ( Someone’s got to fix it… Just pick someone.)
- “No, that’s ”the responsibility of ‘that one developer’” ( The developer in question left the company a year ago.)
- “No, that’s not our problem because we’re being decommissioned in 4 months” ( It’s still your problem until then.)
What's more interesting is when the finding is in their application but somehow the remediation should be done by another team
- “NO, this finding should be remediated by this framework we use, not us.” (Yes, agreed. Until then, it's your responsibility to remediate it unless you get that team to take responsibility.)
- “No, this finding should be remediated by the UI team, not us” ( Yes, output encoding for XSS would be good, but you should also be sanitizing your inputs on the backend…)
- “No, this finding should be remediated by the Cloud team, not us” ( Yes, but until they agree it’s their responsibility, it’s still your app.)
None of this is to say that pushing back about ownership is wrong. The team owns the system should remediate the system and if that’s not the identified team then that’s fine. However as a developer, you should have some idea who the right team should be and if you can point us in the right direction, we’re cool with asking them and getting the right people to work on it.
When it’s all said and done, your penetration testing team doesn’t hate you. However, if it ever seems like they do, here are the top five things you can do to become best friends with your penetration testing team no questions asked.
- Respect your penetration testing team’s time. They likely service a lot of development teams and using their time effectively makes everyone happy.
- Please discuss how they can best give you results prior to testing. If it’s not through reports then tell them before they make the report. ( May still need a report for compliance reasons through…)
- Argue wisely, logically and responsibly. Pushing back is valuable as long as your goal is securing your app and not ignoring the problems.
- Understand that penetration testing is designed to identify issues in applications before their exploited, not shame or attack a development team in any way.
- Ask respectful informed questions. Nothing makes a penetration testing team happier than seeing you’ve ingested their feedback, mulled it over, and come back with relevant questions about the issues at hand.
If you can do all of this, It will go a long to towards making sure you and your penetration testing team never have a fight again.