Measuring the Effectiveness of a Penetration Test
Measuring the effectiveness of a penetration test is tough. Everyone has a different method to determine if a penetration test was “effective”. We recently completed an assessment for a client that came back with over 100 vulnerabilities. They had the exact same penetration test performed the prior year by a different firm and had less than 20 findings. As far as they knew, the prior penetration test had been “effective” and were very surprised by the results of our test.
At the end of the day, you want a report to come out clean with no vulnerabilities, however, this could also lead to paranoia that something could have been missed. Did you receive a thorough test? Did the firm you hired just run a vulnerability scan and call it a day? Did they miss something? Today, we will discuss ways to ensure you are receiving a thorough security assessment and how to manage expectations.
Ways to Measure Effectiveness of a Penetration Test:
- Compare Reports – Have you had a penetration test in prior years? Do you know there are findings from previous reports that have not been remediated? Compare reports to determine if anything is missing. Of course penetration tests can vary from year to year and from tester to tester. Scope can also change with the addition of new systems or the retirement of older systems, significantly changing your security posture, making it even more challenging to make sure you are comparing apples to apples.
- Be Engaged – When reading the final reports and participating in the final presentation, be engaged. Review the results and ask questions to make sure you understand all the findings and the risk they present. What is the risk this finding presents to my firm and how likely is it to be exploited? At Triaxiom, we are happy to answer questions and explain anything we report on further. Any testing organization that is unwilling to present their findings or has difficulty explaining themselves should immediately raise red flags. Additionally, we discuss every finding, why we reported it, ensure it was not a false positive, and then can validate that an implemented fix truly mitigates the risk once it has been remediated.
- Learn and Adapt – Following the penetration test, did you learn anything? If new vulnerabilities were uncovered, did your team notice them during the engagement and would they have been able to thwart a real attack? Partner with the security team to understand the outcomes, implement controls to prevent in the future, and continue to learn.
Ultimately, it is tough to quantify the effectiveness of a penetration test and subsequently compare results. It is not prudent to go into a penetration test and say I expect at least X number of findings, otherwise, it’s going to be a failure. Conversely, you shouldn’t consider a test a failure if you don’t have a completely clean report with nothing to address. Security is constantly evolving and new vulnerabilities and techniques are being discovered. Every penetration test is an opportunity to learn, grow, and mature your security posture so take full advantage of it. Every penetration tester has a different approach and perspective to a test, and if you have 5 different tests by 5 different engineers, odds are each and every report will be different but valuable in some way based on their experience or area of expertise. This is a good thing so embrace this and use it to build and improve your security program.
