Top Mistakes CISOs Make When it Comes to Penetration Testing

Based on working with CISOs for organizations across a variety of verticals, there is a lot they get right about penetration testing as security continues to become more of a focus. But also we see some common mistakes. In this blog, we are going to look into a few of the top mistakes CISOs make when it comes to penetration testing and how your organization can avoid them.

CISO Top Mistakes – #1 – Viewing a Penetration Test as a Reflection of their Job Performance

I would say the top mistake we see is CISOs who believe that a penetration test is a direct reflection of their job performance, as opposed to a helpful tool in their belt to evaluate the effectiveness of security controls in place and to identify any gaps that may exist. Unfortunately, when this happens it can limit the effectiveness of the penetration test. For example, we have encountered some CISOs who will actively try to block or remediate as we are testing, or put us in an isolated network which doesn’t emulate the risk we are trying to evaluate. Moreover, when it is time to go over the results, a major mistake CISOs can make is to be defensive and say things like “Well if this was a real attack, we would have blocked you.” While this may very well be true, and we can conduct different types of tests (such as a red team assessment) to evaluate that, getting defensive can send the message that all the findings are invalid or insignificant, which can have a devastating effect on your organization’s security culture.

To avoid this mistake, the best approach is a conversation with the CISO to explain how this can hamper the results of the test and, ultimately, hurt the organizations security posture. This conversation should note the collaborative nature of a penetration test, and the difference between a red team engagement and penetration test. Additionally, it may help to view a penetration test as an objective evaluation of security controls, rather than the performance of a role. It is likely that the CISO, with unlimited budget and organizational buy in, would have every tool at their disposal and enough resources deployed to lock down the network. Unfortunately, in reality, every organization is limited. A penetration test can help you prioritize your limited resources to get a better return-on-investment when it comes to the effective security provided by the tools and resources you do have.

CISO Top Mistakes – #2 – Limiting the Scope

Another top mistake we see CISOs make is limiting the scope of a test unnecessarily, preventing a more realistic evaluation and a more holistic view of the risks. Frequently, this is due to budget constraints, which may be unavoidable at times. However, it is important to understand that if we are performing a security evaluation on only a subset of the organization, the results will be limited and only provide part of the picture. If you were performing a physical security audit of a bank branch and limited the auditor to just the lobby, it is easy to realize that it doesn’t do you much good if the backdoor is propped open all day. The same concept applies to penetration testing, but it can be harder to visualize. Similarly, when a CISO reaches out for an external penetration test, they should realize that even if the penetration testers were not able to gain access via the network perimeter, a real threat actor would not be under these same constraints and could try to leverage social engineering, the wireless network from the parking lot, or even physically breaking in to obtain a foothold on the network. Unfortunately, many times we also encounter CISOs who “know their users are going to fall for social engineering, so why bother.” While this may be true, it limits their ability to quantify that risk, convey the potential impact to organizational stakeholders, test the effectiveness of technical controls that are in place, or evaluate current security awareness training in managing this risk.

To avoid this mistake, an understanding of the scope is very important. A CISO will always have to balance their budget with their security objectives, and this means that sometimes not everything can be tested. That is fine, as long as the CISO has a good understanding of what was in scope and what residual risk is still present due to portions not being assessed. It is great to walk away from an external penetration test feeling like your network perimeter is relatively secure, however, it is important to keep in mind the other risks that may be present.

CISO Top Mistakes – #3 – Check That Box

The final top mistake we see is organizations that use penetration testing to check a compliance box. They are required to have penetration tests performed to meet a compliance requirement (PCI, SOC 2, etc.) or as part of their cyber insurance policy. The best CISOs realize that they need to make sure to complete the penetration test to meet the requirement, but they don’t stop there. They want to use the penetration test to evaluate their security, to find any gaps, to test the effectiveness of their security tools, etc. In other words, yes it is a requirement, but they are more concerned about the security of their organization and want to use this as a tool in their belt to evaluate their security and make improvements.