Disadvantages of a Bug Bounty Program

In this blog, we are going to talk about some of the disadvantages of a bug bounty program compared to a penetration test. Don’t get us wrong, there are many advantages of a bug bounty program, in fact, we just did an entire blog dedicated to the subject. While Triaxiom Security is a company founded on one-side of that equation (penetration testing), we wanted to present both sides of the argument. So with that said, check out the previous blog for one side of the argument, and here we’ll focus on the other side.

Just as a quick recap, the major differences between a penetration test and a bug bounty program are:

• During a penetration test you will have a highly trained engineer who is dedicated to your project and your project alone for the duration of the assessment.
• Conversely, during a bug bounty program you will have many engineers (thousands, usually) who have the option to test your site, and they will usually be looking for in-depth vulnerabilities that they specialize in.

So now let’s dive into the disadvantages of a bug bounty program.

It’s Not a Comprehensive Test

One of the disadvantages of a bug bounty program is that no one is holistically reviewing your organization, network, or application. During a penetration test you will have a dedicated engineer who is assigned to your project for the duration. Additionally, they will have a specific methodology they will use to methodically review your organization’s testing scope from top to bottom. For an external penetration test for example, this includes open-source reconnaissance, a full port scan, enumerating and evaluating every service, manually attempting exploits, and performing application-level testing on unauthenticated portions of web services.

During a bug bounty, no one has ownership over the project. There are hundreds of testers, but they only get paid if they find a vulnerability. Additionally, they get paid more or less depending on the vulnerability they find. For example, a SQL Injection attack may pay out $500, but an HSTS misconfiguration may only pay out $50. In practical terms, that means you are going to have 99 testers looking for SQL Injection, and hopefully you get one that will check for proper HTTP response configuration. Without ownership over the project, no one can certify that they have looked at every risk, no matter how well it “pays out.” Bug bounties are prone to group-think, where often very obvious vulnerabilities are neglected because everyone assumes someone else has already looked for it.

Trust

It is hard to discuss the disadvantages of a bug bounty program without bringing up the topic of trust. To be fair, bug bounty programs (most of them anyway) have full background checks on the testers they allow into their program. With that being said, they can’t have the same level of trust as a smaller, dedicated firm. At Triaxiom Security for example, we would never hire an engineer, nor put them on a test, without knowing them. I know their family. I have worked with them previously, or at the very least had them referred by a trusted friend. Additionally, before an engineer gets put on a project, they must have at least 5 years of experience. Simply put, I trust every single one of our engineers. With a bug bounty program, they can do some basic level checks, but they don’t know their “employees.”  Their model is similar to Uber, sure, they do some background checks, but they don’t really know the person performing your test. This method works for Uber, but getting a ride to the airport is a little different than allowing someone to hack into your organization.

Control

Another disadvantage of a bug bounty program comes down to control. When we sell a penetration test, we have a kickoff call with the client and go over the entire test, set parameters for the test, and speak to the client about what to focus on, etc. On these calls you will be speaking directly to the engineer who will be on your project. That gives you the opportunity to tell them areas to focus on to support internal initiatives, tell them areas that may cause business disruptions and how to appropriately mitigate that risk, and make sure you are comfortable with all of the rules the engineer will follow. Conversely, with a bug bounty program you will never know, nor speak to the engineers on your project. Sure, you will be able to leave a note, and maybe set a window testing activities, but this isn’t the same as talking directly to the engineer.

Partnership

One of the final disadvantages of a bug bounty program we will bring up in this blog is partnership. One of our core tenets at Triaxiom Security is to partner with our clients. We encourage our clients to reach out to us with any questions before the assessment, after the assessment, or even 6 months down the road. Even if it is not related to the test we perform, we honestly want to help improve your security, and are happy to give any advice we can. Further, the vast majority of our clients have had testing performed by us previously. We have long-standing relationships with our clients and we consider ourselves an extension of their security team. Practically, this means that our reports are customized for their environment. We understand that the marketing department has gone rogue, and there is little you can do to stop it. We understand that your accounting department has that crazy system that still needs Windows XP that you have been trying to get rid of for years. More importantly, we understand that it isn’t practical for your organization to have RSA security tokens that are going to cost you 100K.

Summary

In conclusion, while there are some advantages, there are also many disadvantages of a bug bounty program that must be considered. Simply put, you will not get the same level of trust or control with a bug bounty program than you would from a penetration test. Additionally, bug bounty programs are notorious for group think, where testers just assume other testers have looked at a particular item. Finally, one of the biggest disadvantages of a bug bounty program is that they don’t have a relationship with you. They don’t partner with you over time, and therefore cannot tailor the results to match your organization’s level of risk, security initiatives, or budget.