The first time Katie Paxton-Fear found a bug, she thought it was just luck.
One of her friends had signed her up for an event in London, where hackers aim to find the vulnerabilities in a particular piece of software.
Without any experience of cybersecurity beyond being a programmer and developer, she found one bug, then another. “To be fair, I thought it was a fluke,” she says. But since then she’s found 30 more security bugs.
“It’s kind of like playing Sherlock Holmes,” says Paxton-Fear.
“You feel like a detective, going in rooting around and saying, ‘That looks interesting’, and having a stream of clues,” she says. “And, when you get all the pieces neatly together, and it works and there’s a bug there – it’s the most thrilling experience ever.”
But unlike a hacker looking for vulnerabilities to cause damage or steal data, Paxton-Fear is a bug bounty hunter. The bugs she finds are reported to the companies that write the code.
SEE: Security Awareness and Training policy (TechRepublic Premium)
That allows these organisations to fix the problems before malicious hackers find the same weaknesses. And the bug hunters get paid for each one they find.
As such she’s part of a growing industry that allows security researchers to hack into organisations’ software – with their permission – and then report the weaknesses they discover in return for a financial reward.
It’s a different way of approaching computer security, but one that is proving increasingly popular. One key feature is these security researchers will approach a target from the same perspective as a potential attacker.
In that sense, bug bounty hunters are both the detective Holmes and also at least in part his nemesis, Moriarty, although Paxton-Fear says she sees herself more as Sherlock because by finding the bugs and reporting them, she’s helping improve security.
“I’m doing the right thing,” she says.
Not that doing the right thing takes away the thrill: Paxton-Fear found herself shaking when she wrote up the report to detail her first bug.
Finding mistakes in other people’s work
A number of companies now run their own bug bounty programs, which allow hackers to report the flaws they find in their software. There are also companies that organise these programs for firms that don’t want to run them in-house.
Paxton-Fear says what she describes as the “nice pocket money” that she makes from bug bounties is a motivator – but not the only one: “For me it’s a hobby, but I really enjoy it.”
However, for some hackers, bug bounties can mean big paydays.
According to HackerOne, which organised the events that Paxton-Fear attended and organises bug bounties for big businesses and government agencies, nine hackers have now earned more than $1m each in rewards for spotting vulnerabilities.
Thirteen more have hit $500,000 in lifetime earnings, and 146 hackers have now earned $100,000 each.
Researchers doing their hacking on HackerOne’s platform earned nearly $40m in bounties in 2019. That’s nearly equal to the $82m in bounties the company has paid out on behalf of its customers to date – and that doesn’t take into account corporate bug bounty programs that are also paying out millions a year.
Not bad money for finding mistakes in other people’s work.
SEE: Cybersecurity: This is how much top hackers are earning from bug bounties
Tommy DeVoss is one of those nine million-dollar-earning hackers. He is a reformed blackhat hacker turned bug bounty hunter. DeVoss will hunt for bugs a couple of days a week, looking for things that have changed in the systems he is targeting, and maybe checking old bugs to see if there’s been a change that means the flaw is back again.
“I know the mistakes that get made because I’ve made those mistakes,” says bug bounty hunter, Tommy DeVoss.
Image: TJ STEGE/HackerOne
“The biggest determining factor is the fact I’ve just been doing this for so long and I’ve seen so much stuff. I’ve been a system admin and I’ve been a developer. I know the mistakes that get made because I’ve made those mistakes,” he says.
DeVoss says each of the nine millionaire hackers go after a different type of bug.
“None of us have the same skillset and I think that’s why we’re all able to be successful at the same time, instead of fighting each other for the exact same bugs,” he says.
And while this elite group of high earners is very much the minority. For the vast majority the rewards are much lower; HackerOne said that of the hackers who have found at least one vulnerability, half have earned $1,000 or more. But for some hackers, bug bounties are becoming a handy source of additional financial support.
Considering that hacking is often seen as a shady and mysterious world, there’s actually a lot of data about what bug bounty hackers earn, and what motivates them. And it’s not always about the money.
Explanations for motivations
Nearly a quarter of the security researchers surveyed by HackerOne said their entire income comes from hacking. For more than half, at least 50% of their income comes from hacking. The company said the average bounty paid for a critical vulnerability stood at $3,650, while the average amount paid per vulnerability is $979.
Hacking is a relatively young person’s activity: over 80% are aged under 35 and only half of one percent are over 50. And it’s very male, with only 10% identifying as female or non-binary.
Three-quarters have a degree or postgraduate qualification in computer programming or computer science. Only 14% have no training in the subject at all. However, when it comes to hacking, nearly half describe themselves as self-taught.
Nearly half of hackers describe themselves as self-taught.
Image: Getty Images/iStockphoto
Hackers also earned 38% more in bounty payments in 2019 compared with 2018, according to data from Bugcrowd, another bug bounty program company, which calculates that its hackers prevented $8.9bn in cybercrime by finding and allowing companies to fix bugs that would otherwise have let attackers into their systems.
Among the other data Bugcrowd collected is that hackers it seems are not early risers: 73% do their hacking in the evening and only 13% do any in the morning. Nearly half spend four hours or less working on bugs and only a super-hardcore 8% do more than 30 hours a week.
Hackers seem to find their way to bug bounties via a variety of routes.
Santiago Lopez, another of HackerOne’s elite group of million-dollar-researchers, became intrigued with hacking after he saw the movie Hackers, and earned his first bug bounty in 2016 – when he was aged 16. He went on to become the first hacker on the platform to make a million dollars in bounties.
“Most of all, having the curiosity to want to break stuff and play around will really decide if you’re cut out for hacker life,” he says.
A movie was also behind how Mico Fraxix got interested in computer security, but for a slightly different reason.
He was working as an IT engineer when Sony Pictures was hacked by North Korea, an attack that was probably in response to the studio’s film comedy, The Interview, which was set in the country.
SEE: Network security policy (TechRepublic Premium)
For Fraxix, the incident sparked an interest in the world of computer security. One option, he realised, was to become a penetration tester who would probe the defences of a company, often working for a security consultancy firm. But this path was expensive and demanded a degree in cybersecurity. The second option was to become a bug bounty hunter, and he went on to be one of Bugcrowd’s most successful.
“When I first read online that it’s possible to hack companies and not get prosecuted for it, I was thrilled and amazed,” he says and worked full time as a bug bounty hunter before moving on to a job in penetration testing – and paying for the training through bug hunting.
So what makes a good bug bounty hunter? Paxton-Fear reckons being a developer is a big advantage.
“I have an innate sense of how I would do it and I assume people think like me,” she says.
“One of the big skills in bug bounties that’s really difficult to teach is intuition. Everything I do I am following my intuition. It’s what looks interesting and what doesn’t look right.”
Big rewards for helping big tech
Bug bounty programs have actually been around for a long time. Browser pioneer Netscape launched the first one back in 1995. A few years later, Mozilla decided to launch a similar program to allow users to report bugs in its software – a program that still runs today.
Mozilla started out with enough money for 10 bounties but didn’t know whether the idea was going to take off or not.
“We are the oldest security bug bounty that’s still operating,” says Daniel Veditz, senior staff security engineer at Mozilla. “We were a small company and it seemed a good way to encourage people to look into security problems.”
But from modest beginnings, Mozilla’s bug bounty program has grown. Between 2017 and 2019, Mozilla paid out nearly a million dollars – $965,750 to be precise – to researchers who reported 348 bugs, with an average payout of $2,775 per bug. The Firefox browser maker will pay between $3,000 and $10,000 to researchers who spot potentially exploitable critical and high-rated client security vulnerabilities.
But for Veditz, having a bug bounty program is also a signal about a company’s attitude towards security. It shows that the company welcomes security researchers and sees value in their work. “We want to send a signal – we care, please come bother. If you’ve found something it helps everyone out.”
And, after Netscape and Mozilla’s early experiments, many other big tech companies followed. Now bug bounties are offered on anything from bugs in websites to cloud services, business software or mobile apps.
“We started it as an experiment and there was no one around to encourage us or compare ourselves to,” says Veditz. “Along the way lots of other people have decided that it’s a good idea and emulated us and surpassed us in the amount of money they can afford to pay folks.”
Among those big spenders on bug bounties are some of the biggest tech giants. Microsoft now offers rewards to security researchers who find vulnerabilities across a range of its products, from Microsoft Azure to Xbox, Microsoft Dynamics 365 to its new Edge browser.
SEE: Facebook launches bug bounty ‘loyalty program’
Earlier this year Microsoft said it had spent $13.7m in bounties in the past 12 months – over three times the $4.4m it spent in the year before. That’s a big number, but so are the potential awards to individuals. A researcher who discovers a critical remote code execution, information disclosure, or denial-of-service vulnerability in Microsoft’s Hyper-V could earn up to $250,000, while vulnerability reports on Microsoft Azure cloud services could earn $40,000.
Microsoft also noted that, with many unable to leave their homes due to COVID-19 lockdowns, bug bounty hunters have been busy. Across all 15 of its bounty programs, it saw a rise in bug reports during the first several months of the pandemic.
Google is another big spender on bug bounties, spending a total of $21m since it launched its vulnerability reward programs a decade ago, including $6.5m in 2019 – twice what it had previously paid out in a single year.
It also has some huge potential bounties on offer, with a top prize of $1m for a “full chain remote code execution exploit with persistence” which compromises the Titan M secure chip on Pixel devices. And if the exploit is on specific developer preview versions of Android, there’s a 50% bonus, taking the reward up to a cool $1.5m.
After these giants kicked off bug bounty programs, many other tech companies saw the benefits of the approach, making it a common option. But in recent years, the vogue for bug bounties has spread beyond tech – now many large businesses provide some kind of reward. That’s largely thanks to the US Department of Defense, which launched its Hack the Pentagon in 2016 as the federal government’s first bug bounty program, which since then has allowed it to identify – and fix – thousands of security vulnerabilities across public-facing systems.
Getting eyes on the prize
So why does code have flaws in the first place?
Part of the issue is the way that software is written. It’s usually written in a hurry, with a deadline looming and the boss pacing up and down. It’s written by multiple teams with slightly different experiences and different skills and priorities. Those teams will then have to somehow merge those projects together and make sure the end result is secure.
“It’s not that there are fewer bugs, it’s just the bugs are in different places,” says Katie Paxton-Fear.
Image: HackerOne
But then, most likely, the objectives of the project will shift and a new feature is needed, which means new code being added on top. And then, maybe a year or two later, long after the original development team has moved on, a feature will need changing or removing, which means a new team of developers trying to understand, then modify, the whole leaning tower of code. And this is the best-case scenario for development in many cases. No wonder hackers find gaps they can sneak through.
Paxton-Fear says part of the problem is that software development is so complex and involves multiple teams.
“You have all kinds of different developers who touch a piece of software. You get development time that is often really squished for a feature. As a developer you just want to push features out on time. You’re passing code around and little things could be missed all the time – it’s just unfortunate some of these end up being huge security risks,” she says.
The benefits for the researchers are the chance to poke around in other peoples’ systems – something usually frowned on at best – while getting paid and maybe becoming a hacker celebrity.
SEE: FireEye’s bug bounty program goes public
For the companies that use bug bounty programs, the benefit comes from being able to get lots of seasoned hackers to look at their code in exactly the same way that attackers would – but without the risk – and to pay up only if they find anything.
GitLab launched a private vulnerability disclosure program in 2014 and has since moved on to a public program with HackerOne. It has now paid out a million dollars across 768 bug reporters.
“The main value we get from it is reducing risk – that’s the ultimate goal,” says James Ritchey, manager of app security at GitLab. “To do that we need to be aware of our security issues – and what better way to do that than having more eyes on the product. It helps our security team scale.”
It’s also an acknowledgment of the reality of computer security and the threat that every organisation faces when they have systems exposed to the internet.
“Ignorance isn’t bliss in security, so we really want to know about these security issues and all those eyes can give us a better perspective. The truth is the moment you’re on the internet, you are kind of an open target anyway. At that point it’s better to have a financial outing for those hackers because they’re going to hack anyway,” he says.
Turning a hobby into a career
Prash Somaiya, technical program manager at HackerOne, says the bug bounty programs it organises give companies access to skills they couldn’t easily access otherwise. Some companies have such sprawling infrastructure that it’s hard for them to even understand where their own systems are – let alone testing them for security.
He says the key difference between hiring consultants to do penetration testing and setting up a bug bounty program is that researchers aren’t being paid for their time, and you’re not paying an hourly rate for them to find bugs – it’s all about delivering results.
“Security is an evolving beast. Every organisation has vulnerabilities present in their software no matter what, and it’s about acknowledging that and working with the security community to uncover these flaws,” he says. “If those vulnerabilities are out there on the internet, they can be found and they can be exploited.”
Security is an ever-evolving beast.
Image: Getty Images/iStockphoto
However, a bug bounty program isn’t a replacement for more traditional forms of security testing, but an addition, cautions Mozilla’s Veditz: “There are companies that jump into a bug bounty program thinking that it’s a substitute for quality assurance or testing or a security program – and that’s a road to disaster.”
Some critics warn that bug programs are being used as a sticking plaster when actually organisations need to fundamentally rethink how they write code. They say companies should not be relying on outsiders – many are self-taught and doing it for fun, or working in lower-cost economies where the money from bounties goes further – to fix basic errors that in-house teams should have spotted themselves.
They argue that companies should ensure their internal development processes encourage secure coding rather than adding security in as an afterthought, or hoping that external hackers can fix the problem later.
SEE: Bug bounty platform ZDI awarded $25m to researchers over the past 15 years
Taking into account the additional developer time, the cost of the bug bounty program and the cost of any potential security breaches in the interim, making sure the code is secure before it is published is always going to be much cheaper than fixing it later.
In addition, to set up a bug bounty program without having the developers in place who can actually trace and fix the bugs discovered – which is a very different skill to finding them in the first place – means that security is unlikely to be improved as a result. It might even make things worse by creating a false sense of security.
Indeed, bug bounty programs are not the answer to every problem, and can create some of their own. Some researchers do not want to be involved in them because some programs limit their ability to share the vulnerabilities that they discover, something that would be a benefit to all users of that particular software, and also help them build their own reputation.
There’s also a broader criticism of the model – that, like many other crowdsourcing models, the rewards are hard to earn. There are relatively few hackers who make big money.
This economic pressure is perhaps part of the reason behind the geographic spread of researchers chasing bug bounties. For Bugcrowd, 80% of bounties are from US companies, but 34% are paid out to Indian researchers – compared with 26% that go to US researchers.
For HackerOne, nearly 90% of bounties come from the US, and while US hackers get the most, researchers from India, Russia and China also do well. That means bug bounties could in some respects evolve into a crowdsourced twist on the established model of offshore outsourcing.
Paying by results keeps costs down, but may also encourage researchers to focus on easier-to-spot flaws they can dig out using automated tools, rather than the ones that might take significantly more time and effort, further creating a false sense of security.
And it’s also worth remembering that for most participants, bug hunting is a fun pastime. Some may wonder whether it is wise for the largest organisations in the world to rely on hobbyists for their online security.
More positively, many hackers see proving their prowess as bug hunters as a route into the security industry, which is desperate for talent. If bug bounties can demonstrate they have a role in creating an on-ramp for new security professionals – as they did for Fraxix – then some of the criticism may go away.
Hacking is a team sport
One thing that might surprise outsiders is the amount of cooperation between hackers. Even though only one of them is ever going to be able to claim any particular bounty, the bug bounty hacker community openly shares most information, says DeVoss.
“One of the major parts of becoming good when it comes to hacking and bug bounties is there are always going to be people smarter than you, who know more than you or who know different things than you,” he says. “I do this for the money but I’m not greedy, so I don’t mind other people making money as well.”
Paxton-Fear agrees: “I know that if I have a problem I can ask 10 different people for help and rely on their expertise, and a lot of the time they won’t ask for money back – they just want to help. Everyone realises what it was like to get started.”
Image: Hacking is a community and bug bounties are now part of the mainstream.
Getty Images/iStockphoto
Bug bounties have come a long way since the day of Netscape’s first experiment. They’re now firmly part of the mainstream of the security industry. So as the number of wannabe hackers – and companies comfortable with employing them – increases, how does that change the bug bounty world?
“Hacking will always be a good opportunity for people who don’t want to follow a traditional corporate career path and want the flexibility that comes with the territory,” says Lopez, adding that as awareness of bug bounty hacking grows, it will certainly become less niche, which means more competition.
SEE: Microsoft goes big in security bug bounties: Its $13.7m is double Google’s 2019 payouts
Developers are also wising up, which means that some of the easiest bugs are now harder to find.
Companies have matured drastically over the past few years, says Fraxix: “It used to be that you could easily compromise famous brands and companies but, nowadays, it’s a lot more difficult. Companies are better prepared and their development teams are better trained.”
That’s especially true when organisations have been running bug bounties for a while.
GitLab’s Ritchey says when it first ran the program, there were very straightforward findings that were very easy to reproduce.
“Nowadays, it’s much more complex. The thing is we are constantly releasing new features and updating our own software, and because of that the security issues will never go away. Security issues will always be there – the important thing is to have a multilayered approach to it.”
The best defence against the worst problem
And for sure, the types of vulnerabilities being hunted have changed. When the first bug bounties were launched, the cloud and smartphones didn’t exist. Yet those areas are where some of the biggest bounties can now be scored.
But that focus may prove to be a mismatch for the bug bounty business model, because most hackers concentrate on web security rather than these more complicated areas that often require additional skills and experience. In Bugcrowd’s latest research for example, 70% of hackers listed web application testing skills, but only 3% listed Android app skills.
Still, nobody is seriously expecting computer security to improve to the point at which bug bounties – or all the other techniques used to test code once it has been written – are going to be retired any time soon.
“I don’t think it’s going to get harder… I think it’s going to get different,” says Paxton-Fear. While a few years back bug hunting was all about particular flaws like cross-site scripting and JavaScript injections, now developers know about these problems.
But thanks to the Internet of Things, the number of devices with some kind of computing power being connected continues to expand, which means new and unusual targets for researchers, like an internet-connected fridge.
“There are bugs in your fridge for sure,” says Paxton-Fear. “There’s not this ending where developers suddenly know every bug – it just changes. It’s not that there are fewer bugs, it’s just the bugs are in different places.”
SEE: IoT: Major threats and security tips for devices (free PDF) (TechRepublic)
Mozilla’s Veditz agrees; hackers find bugs because they come to the code with that outsider approach, and that’s not going to change.
“As long as there are bugs in software, there are security bugs, and somebody’s got to find them. Bug bounties are a good way to encourage an outside look. Bug bounties as a concept are here to stay for the foreseeable future until we get perfect robots writing our code that don’t make mistakes.”
Even perfect robots are unlikely to make bug bounty hunters redundant according to DeVoss, who argues there is no such thing as a 100%-secured computer system – unless that computer is turned off.
Because of the way that software is written – over years in some cases by teams contributing different elements and adding new features over time – code that seems secure at one point may develop problems as it is altered at a later date.
“As long as we still have humans writing the code, there’re going to be errors. And even when we get to where AI starts writing code and finding bugs, they’re still going to be there. Just because something seems secure today doesn’t mean that in a month, six months, a year, or five years from now, something is found that completely breaks it all”, he says.
Lopez has a similar view; don’t expect AIs writing perfect code to put smart humans out of business, he says: “There’ll always be a need for hackers. Even with AI and security built in from the outset, there will always be people who want to break stuff and who will learn to manipulate AI to do so. Human hackers are the best defense against the most sophisticated attacks.”
