For more than 15 years, Google Safe Browsing has been protecting users from phishing, malware, unwanted software and more, by identifying and warning users about potentially abusive sites on more than 5 billion devices around the world. As attackers grow more sophisticated, we've seen the need for protections that can adapt as quickly as the threats they defend against. That’s why we're excited to announce a new version of Safe Browsing that will provide real-time, privacy-preserving URL protection for people using the Standard protection mode of Safe Browsing in Chrome.
Chrome automatically protects you by flagging potentially dangerous sites and files, hand in hand with Safe Browsing which discovers thousands of unsafe sites every day and adds them to its lists of harmful sites and files.
So far, for privacy and performance reasons, Chrome has first checked sites you visit against a locally-stored list of known unsafe sites which is updated every 30 to 60 minutes – this is done using hash-based checks.
Hash-based check overview
But unsafe sites have adapted — today, the majority of them exist for less than 10 minutes, meaning that by the time the locally-stored list of known unsafe sites is updated, many have slipped through and had the chance to do damage if users happened to visit them during this window of opportunity. Further, Safe Browsing’s list of harmful websites continues to grow at a rapid pace. Not all devices have the resources necessary to maintain this growing list, nor are they always able to receive and apply updates to the list at the frequency necessary to benefit from full protection.
Safe Browsing’s Enhanced protection mode already stays ahead of such threats with technologies such as real-time list checks and AI-based classification of malicious URLs and web pages. We built this mode as an opt-in to give users the choice of sharing more security-related data in order to get stronger security. This mode has shown that checking lists in real time brings significant value, so we decided to bring that to the default Standard protection mode through a new API – one that doesn't share the URLs of sites you visit with Google.
In order to transition to real-time protection, checks now need to be performed against a list that is maintained on the Safe Browsing server. The server-side list can include unsafe sites as soon as they are discovered, so it is able to capture sites that switch quickly. It can also grow as large as needed because the Safe Browsing server is not constrained in the same way that user devices are.
Behind the scenes, here's what is happening in Chrome:
In order to preserve user privacy, we have partnered with Fastly, an edge cloud platform that provides content delivery, edge compute, security, and observability services, to operate an Oblivious HTTP (OHTTP) privacy server between Chrome and Safe Browsing – you can learn more about Fastly's commitment to user privacy on their Customer Trust page. With OHTTP, Safe Browsing does not see your IP address, and your Safe Browsing checks are mixed amongst those sent by other Chrome users. This means Safe Browsing cannot correlate the URL checks you send as you browse the web.
Before hash prefixes leave your device, Chrome encrypts them using a public key from Safe Browsing. These encrypted hash prefixes are then sent to the privacy server. Since the privacy server doesn’t know the private key, it cannot decrypt the hash prefixes, which offers privacy from the privacy server itself.
The privacy server then removes potential user identifiers such as your IP address and forwards the encrypted hash prefixes to the Safe Browsing server. The privacy server is operated independently by Fastly, meaning that Google doesn’t have access to potential user identifiers (including IP address and User Agent) from the original request. Once the Safe Browsing server receives the encrypted hash prefixes from the privacy server, it decrypts the hash prefixes with its private key and then continues to check the server-side list.
Ultimately, Safe Browsing sees the hash prefixes of your URL but not your IP address, and the privacy server sees your IP address but not the hash prefixes. No single party has access to both your identity and the hash prefixes. As such, your browsing activity remains private.
Real-time check overview
Compared with the hash-based check, the real-time check requires sending a request to a server, which adds additional latency. We have employed a few techniques to make sure your browsing experience continues to be smooth and responsive.
First, before performing the real-time check, Chrome checks against a global and local cache on your device to avoid unnecessary delay.
Both caches are stored in memory, so it is much faster to check them than sending a real-time request over the network.
In addition, Chrome follows a fallback mechanism in case of unsuccessful or slow requests. If the real-time request fails consecutively, Chrome will enter a back-off mode and downgrade the checks to hash-based checks for a certain period.
We are also in the process of introducing an asynchronous mechanism, which will allow the site to load while the real-time check is in progress. This will improve the user experience, as the real-time check won’t block page load.
With the latest release of Chrome for desktop, Android, and iOS, we’re upgrading the Standard protection mode of Safe Browsing so it will now check sites using Safe Browsing’s real-time protection protocol, without sharing your browsing history with Google. You don't need to take any action to benefit from this improved functionality.
If you want more protection, we still encourage you to turn on the Enhanced protection mode of Safe Browsing. You might wonder why you need enhanced protection when you'll be getting real-time URL protection in Standard protection – this is because in Standard protection mode, the real-time feature can only protect you from sites that Safe Browsing has already confirmed to be unsafe. On the other hand, Enhanced protection mode is able to use additional information together with advanced machine learning models to protect you from sites that Safe Browsing may not yet have confirmed to be unsafe, for example because the site was only very recently created or is cloaking its true behavior to Safe Browsing’s detection systems.
Enhanced protection also continues to offer protection beyond real-time URL checks, for example by providing deep scans for suspicious files and extra protection from suspicious Chrome extensions.
The real-time feature of the Standard protection mode of Safe Browsing is on by default for Chrome. If needed, it may be configured using the policy SafeBrowsingProxiedRealTimeChecksAllowed. It is also worth noting that in order for this feature to work in Chrome, enterprises may need to explicitly allow traffic to the Fastly privacy server. If the server is not reachable, Chrome will downgrade the checks to hash-based checks.
While Chrome is the first surface where these protections are available, we plan to make them available to eligible developers for non-commercial use cases via the Safe Browsing API. Using the API, developers and privacy server operators can partner to better protect their products’ users from fast-moving malicious actors in a privacy-preserving manner. To learn more, keep an eye out for our upcoming developer documentation to be published on the Google for Developers site.
Last year, we again witnessed the power of community-driven security efforts as researchers from around the world contributed to help us identify and address thousands of vulnerabilities in our products and services. Working with our dedicated bug hunter community, we awarded $10 million to our 600+ researchers based in 68 countries.
New Resources and Improvements
Just like every year, 2023 brought a series of changes and improvements to our vulnerability reward programs:
As in past years, we are sharing our 2023 Year in Review statistics across all of our programs. We would like to give a special thank you to all of our dedicated researchers for their continued work with our programs - we look forward to more collaboration in the future!
Android and Google Devices
In 2023, the Android VRP achieved significant milestones, reflecting our dedication to securing the Android ecosystem. We awarded over $3.4 million in rewards to researchers who uncovered remarkable vulnerabilities within Android and increased our maximum reward amount to $15,000 for critical vulnerabilities. We also saw a sharpened focus on higher severity issues as a result of our changes to incentivize report quality and increasing rewards for high and critical severity issues.
Expanding our program’s scope, Wear OS has been added to the program to further incentivize research in new wearable technology to ensure users’ safety.
Working closely with top researchers at the ESCAL8 conference, we also hosted a live hacking event for Wear OS and Android Automotive OS which resulted in $70,000 rewarded to researchers for finding over 20 critical vulnerabilities!
We would also like to spotlight the hardwear.io security conferences. Hardwear.io gave us a platform to engage with top hardware security researchers who uncovered over 50 vulnerabilities in Nest, Fitbit, and Wearables, and received a total of $116,000 last year!
The Google Play Security Reward Program continued to foster security research across popular Android apps on Google Play.
A huge thank you to the researchers who made our program such a success. A special shout out to Zinuo Han (@ele7enxxh) of OPPO Amber Security Lab and Yu-Cheng Lin (林禹成) (@AndroBugs) for your hard work and continuing to be some of the top researchers contributing to Android VRPs!
Chrome
2023 was a year of changes and experimentation for the Chrome VRP. In Chrome Milestone 116, MiraclePtr was launched across all Chrome platforms. This resulted in raising the difficulty of discovery of fully exploitable non-renderer UAFs in Chrome and resulted in lower reward amounts for MiraclePtr-protected UAFs, as highly mitigated security bugs. While code and issues protected by MiraclePtr are expected to be resilient to the exploitation of non-renderer UAFs, the Chrome VRP launched the MiraclePtr Bypass Reward to incentivize research toward discovering potential bypasses of this protection.
The Chrome VRP also launched the Full Chain Exploit Bonus, offering triple the standard full reward amount for the first Chrome full-chain exploit reported and double the standard full reward amount for any follow-up reports. While both of these large incentives have gone unclaimed, we are leaving the door open in 2024 for any researchers looking to take on these challenges.
In 2023, Chrome VRP also introduced increased rewards for V8 bugs in older channels of Chrome, with an additional bonus for bugs existing before M105. This resulted in a few very impactful reports of long-existing V8 bugs, including one report of a V8 JIT optimization bug in Chrome since at least M91, which resulted in a $30,000 reward for that researcher.
All of this resulted in $2.1M in rewards to security researchers for 359 unique reports of Chrome Browser security bugs. We were also able to meet some of our top researchers from previous years who were invited to participate in bugSWAT as part of Google’s ESCAL8 event in Tokyo in October. We capped off the year by publicly announcing our 2023 Top 20 Chrome VRP reporters who received a bonus reward for their contributions.
Thank you to the Chrome VRP security researcher community for your contributions and efforts to help us make Chrome more secure for everyone!
Generative AI
Last year, we also ran a bugSWAT live-hacking event targeting LLM products. Apart from fun, sun, and a lot to do, we also got 35 reports, totaling more than $87,000 - and discovered issues like Johann, Joseph, and Kai’s “Hacking Google Bard - From Prompt Injection to Data Exfiltration” and Roni, Justin, and Joseph’s “We Hacked Google A.I. for $50,000”.
To help AI-focused bughunters know what’s in scope and what’s not, we recently published our criteria for bugs in AI products. This criteria aims to facilitate testing for traditional security vulnerabilities as well as risks specific to AI systems, and is one way that we are implementing the voluntary AI commitments that Google made at the White House in July.
Looking Forward
We remain committed to fostering collaboration, innovation, and transparency with the security community. Our ongoing mission is to stay ahead of emerging threats, adapt to evolving technologies, and continue to strengthen the security posture of Google’s products and services. We look forward to continuing to drive greater advancements in the world of cybersecurity.
A huge thank you to our bug hunter community for helping to make Google products and platforms more safe and secure for our users around the world!
Thank you to Adam Bacchus, Dirk Göhmann, Eduardo Vela, Sarah Jacobus, Amy Ressler, Martin Straka, Jan Keller, Tony Mendez.
Google’s Project Zero reports that memory safety vulnerabilities—security defects caused by subtle coding errors related to how a program accesses memory—have been "the standard for attacking software for the last few decades and it’s still how attackers are having success". Their analysis shows two thirds of 0-day exploits detected in the wild used memory corruption vulnerabilities. Despite substantial investments to improve memory-unsafe languages, those vulnerabilities continue to top the most commonly exploited vulnerability classes.
In this post, we share our perspective on memory safety in a comprehensive whitepaper. This paper delves into the data, challenges of tackling memory unsafety, and discusses possible approaches for achieving memory safety and their tradeoffs. We'll also highlight our commitments towards implementing several of the solutions outlined in the whitepaper, most recently with a $1,000,000 grant to the Rust Foundation, thereby advancing the development of a robust memory-safe ecosystem.
2022 marked the 50th anniversary of memory safety vulnerabilities. Since then, memory safety risks have grown more obvious. Like others', Google's internal vulnerability data and research show that memory safety bugs are widespread and one of the leading causes of vulnerabilities in memory-unsafe codebases. Those vulnerabilities endanger end users, our industry, and the broader society. We're encouraged to see governments also taking this issue seriously, as with the U.S. Office of the National Cyber Director publication of a paper on the topic last week.
By sharing our insights and experiences, we hope to inspire the broader community and industry to adopt memory-safe practices and technologies, ultimately making technology safer.
At Google, we have decades of experience addressing, at scale, large classes of vulnerabilities that were once similarly prevalent as memory safety issues. Our approach, which we call “Safe Coding”, treats vulnerability-prone coding constructs themselves as hazards (i.e., independently of, and in addition to, the vulnerability they might cause), and is centered around ensuring developers do not encounter such hazards during regular coding practice.
Based on this experience, we expect that high assurance memory safety can only be achieved via a Secure-by-Design approach centered around comprehensive adoption of languages with rigorous memory safety guarantees. As a consequence, we are considering a gradual transition towards memory-safe languages like Java, Go, and Rust.
Over the past decades, in addition to large Java and Go memory-safe codebases, Google has developed and accumulated hundreds of millions of lines of C++ code that is in active use and under active, ongoing development. This very large existing codebase results in significant challenges for a transition to memory safety:
We see no realistic path for an evolution of C++ into a language with rigorous memory safety guarantees that include temporal safety.
A large-scale rewrite of all existing C++ code into a different, memory-safe language appears very difficult and will likely remain impractical.
We consider it important to complement a transition to memory safe languages for new code and particularly at-risk components with safety improvements for existing C++ code, to the extent practicable. We believe that substantial improvements can be achieved through an incremental transition to a partially-memory-safe C++ language subset, augmented with hardware security features when available. For instance, see our work improving spatial safety in GCP's networking stack.
We are actively investing in many of the solutions outlined in our whitepaper and in our response to the US Federal Government’s RFI on Open Source Software Security.
Android has written several components in Rust over the last few years, leading to compelling security improvements. In Android’s Ultra-wideband (UWB) module, this has improved the security of the module while also reducing the memory usage and inter-procedural calls.
Chrome has started shipping some features in Rust; in one case, Chrome was able to move its QR code generator out of a sandbox by adopting a new memory-safe library written in Rust, leading to both better security and better performance.
Google recently announced a $1,000,000 grant to the Rust foundation to enhance interoperability with C++ code. This will facilitate incremental adoption of Rust in existing memory-unsafe code bases, which will be key to enabling even more new development to occur in a memory-safe language. Relatedly, we are also working on addressing cross-language attacks that can occur when mixing Rust and C++ in the same binary.
Google is investing in building the memory-safe open-source ecosystem through ISRG Prossimo and OpenSSF’s Alpha-Omega project. Back in 2021, we funded efforts to bring Rust to the Linux Kernel, which is now enabling us to write memory-safe drivers. This funding is also going towards providing alternatives or upgrades to key open-source libraries in a memory-safe language, such as providing a memory safe TLS implementation.
We know that memory safe languages will not address every security bug, but just as our efforts to eliminate XSS attacks through tooling showed, removing large classes of exploits both directly benefits consumers of software and allows us to move our focus to addressing further classes of security vulnerabilities.
To access the full whitepaper and learn more about Google's perspective on memory safety, visit https://research.google/pubs/secure-by-design-googles-perspective-on-memory-safety/
From its founding, Android has been guided by principles of openness, transparency, safety, and choice. Android gives you the freedom to choose which device best fits your needs, while also providing the flexibility to download apps from a variety of sources, including preloaded app stores such as the Google Play Store or the Galaxy Store; third-party app stores; and direct downloads from the Internet.Keeping users safe in an open ecosystem takes sophisticated defenses. That’s why Android provides multiple layers of protections, powered by AI and backed by a large dedicated security & privacy team, to help to protect our users from security threats while continually making the platform more resilient. We also provide our users with numerous built-in protections like Google Play Protect, the world’s most widely deployed threat detection service, which actively scans over 125 billion apps on devices every day to monitor for harmful behavior. That said, our data shows that a disproportionate amount of bad actors take advantage of select APIs and distribution channels in this open ecosystem.
Elevating app security in an open ecosystem
While users have the flexibility to download apps from many sources, the safety of an app can vary depending on the download source. Google Play, for example, carries out rigorous operational reviews to ensure app safety, including proper high-risk API use and permissions handling. Other app stores may also follow established policies and procedures that help reduce risks to users and their data. These protections often include requirements for developers to declare which permissions their apps use and how developers plan to use app data. Conversely, standalone app distribution sources like web browsers, messaging apps or file managers – which we commonly refer to as Internet-sideloading – do not offer the same rigorous requirements and operational reviews. Our data demonstrates that users who download from these sources today face unusually high security risks due to these missing protections.
We recently launched enhanced Google Play Protect real-time scanning to help better protect users against novel malicious Internet-sideloaded apps. This enhancement is designed to address malicious apps that leverage various methods, such as AI, to avoid detection. This feature, now deployed on Android devices with Google Play Services in India, Thailand, Singapore and Brazil, has already made a significant impact on user safety.
As a result of the real-time scanning enhancement, Play Protect has identified 515,000 new malicious apps and issued more than 3.1 million warnings or blocks of those apps. Play Protect is constantly improving its detection capabilities with each identified app, allowing us to strengthen our protections for the entire Android ecosystem.
A new pilot to combat financial fraud
Cybercriminals continue to invest in advanced financial fraud scams, costing consumers more than $1 trillion in losses. According to the 2023 Global State of Scams Report by the Global Anti-Scam Alliance, 78 percent of mobile users surveyed experienced at least one scam in the last year. Of those surveyed, 45 percent said they’re experiencing more scams in the last 12 months. The Global Scam Report also found that scams were most often initiated by sending scam links via various messaging platforms to get users to install malicious apps and very often paired with a phone call posing to be from a valid entity.
Scammers frequently employ social engineering tactics to deceive mobile users. Using urgent pretenses that often involve a risk to a user’s finances or an opportunity for quick wealth, cybercriminals convince users to disable security safeguards and ignore proactive warnings for potential malware, scams, and phishing. We’ve seen a large percentage of users ignore, or are tricked into dismissing, these proactive Android platform warnings and proceed with installing malicious apps. This can lead to users ultimately disclosing their security codes, passwords, financial information and/or transferring funds unknowingly to a fraudster.
To help better protect Android users from these financial fraud attacks, we are piloting enhanced fraud protection with Google Play Protect. As part of a continued strategic partnership with the Cyber Security Agency of Singapore (CSA), we will launch this first pilot in Singapore in the coming weeks to help keep Android users safe from mobile financial fraud.
This enhanced fraud protection will analyze and automatically block the installation of apps that may use sensitive permissions frequently abused for financial fraud when the user attempts to install the app from an Internet-sideloading source (web browsers, messaging apps or file managers). This enhancement will inspect the permissions the app declared in real-time and specifically look for four permission requests: RECEIVE_SMS, READ_SMS, BIND_Notifications, and Accessibility. These permissions are frequently abused by fraudsters to intercept one-time passwords via SMS or notifications, as well as spy on screen content. Based on our analysis of major fraud malware families that exploit these sensitive permissions, we found that over 95 percent of installations came from Internet-sideloading sources.
During the upcoming pilot, when a user in Singapore attempts to install an application from an Internet-sideloading source and any of these four permissions are declared, Play Protect will automatically block the installation with an explanation to the user.
Collaborating to combat mobile fraud
This enhanced fraud protection has undergone testing by the Singapore government and will be rolling out to Android devices with Google Play services.
“The fight against online scams is a dynamic one. As cybercriminals refine their methods, we must collaborate and innovate to stay ahead, “ said Mr Chua Kuan Seah, Deputy Chief Executive of CSA. “Through such partnerships with technology players like Google, we are constantly improving our anti-scam defenses to protect Singaporeans online and safeguard their digital assets.”
Together with CSA, we will be closely monitoring the results of the pilot program to assess its impact and make adjustments as needed. We will also support CSA by continuing to assist with malware detection and analysis, sharing malware insights and techniques, and creating user and developer education resources.
How developers can prepareFor developers distributing apps that may be affected by this pilot, please take the time to review the device permissions your app is requesting and ensure you’re following developer best practices. Your app should only request permissions that the app needs to complete an action and ensure it does not violate the Mobile Unwanted Software principles. Always ensure that your app does not engage in behavior that could be considered potentially harmful or malware.
If you find that your app is affected by the app protection pilot you can refer to our updated developer guidance for Play Protect warnings for tips on how to help fix potential issues with your app and instructions for filing an appeal if needed.
Check out the video below to learn more.
Our commitment to protecting Android users
We believe industry collaboration is essential to protect users from mobile security threats and fraud. Piloting these new protections will help us stay ahead of new attacks and evolve our solutions to defeat scammers and their expanding fraud attempt. We have an unwavering commitment to protecting our users around the world and look forward to continuing to partner with governments, ecosystem partners and other stakeholders to improve user protections.
Back in 2021, we announced that Google was joining the Rust Foundation. At the time, Rust was already in wide use across Android and other Google products. Our announcement emphasized our commitment to improving the security reviews of Rust code and its interoperability with C++ code. Rust is one of the strongest tools we have to address memory safety security issues. Since that announcement, industry leaders and government agencies have echoed our sentiment.
We are delighted to announce that Google has provided a grant of $1 million to the Rust Foundation to support efforts that will improve the ability of Rust code to interoperate with existing legacy C++ codebases. We’re also furthering our existing commitment to the open-source Rust community by aggregating and publishing audits for Rust crates that we use in open-source Google projects. These contributions, along with our previous interoperability contributions, have us excited about the future of Rust.
“Based on historical vulnerability density statistics, Rust has proactively prevented hundreds of vulnerabilities from impacting the Android ecosystem. This investment aims to expand the adoption of Rust across various components of the platform.” – Dave Kleidermacher, Google Vice President of Engineering, Android Security & Privacy
While Google has seen the most significant growth in the use of Rust in Android, we’re continuing to grow its use across more applications, including clients and server hardware.
“While Rust may not be suitable for all product applications, prioritizing seamless interoperability with C++ will accelerate wider community adoption, thereby aligning with the industry goals of improving memory safety.” – Royal Hansen, Google Vice President of Safety & Security
The Rust tooling and ecosystem already support interoperability with Android and with continued investment in tools like cxx, autocxx, bindgen, cbindgen, diplomat, and crubit, we are seeing regular improvements in the state of Rust interoperability with C++. As these improvements have continued, we’ve seen a reduction in the barriers to adoption and accelerated adoption of Rust. While that progress across the many tools continues, it is often only expanded incrementally to support the particular needs of a given project or company.
In order to accelerate both Rust adoption at Google as well as more broadly across the industry, we are eager to invest in and collaborate on any needed ABI changes, tooling and build system support, wrapper libraries, or other areas identified.
We are excited to support this work through the Rust Foundation’s Interop Initiative and in collaboration with the Rust project to ensure that any additions made are suitable and address the challenges of Rust adoption that projects using C++ face. Improving memory safety across the software industry is one of the key technology challenges of our time, and we invite others across the community and industry to join us in working together to secure the open source ecosystem for everyone.
Learn more about the Rust Foundation’s Interop Initiative by reading their recent announcement.
This week, the United Nations convened member states to continue its years-long negotiations on the UN Cybercrime Treaty, titled “Countering the Use of Information and Communications Technologies for Criminal Purposes.”
As more aspects of our lives intersect with the digital sphere, law enforcement around the world has increasingly turned to electronic evidence to investigate and disrupt criminal activity. Google takes the threat of cybercrime very seriously, and dedicates significant resources to combating it. When governments send Google legal orders to disclose user data in connection with their investigations, we carefully review those orders to make sure they satisfy applicable laws, international norms, and Google’s policies. We also regularly report the number of these orders in our Transparency Report.
To ensure that transnational legal demands are issued consistent with rule of law, we have long called for an international framework for digital evidence that includes robust due process protections, respects human rights (including the right to free expression), and aligns with existing international norms. This is particularly important in the case of transnational criminal investigations, where the legal protections in one jurisdiction may not align with those in others.
Such safeguards aren’t just important to ensuring free expression and human rights, they are also critical to protecting web security. Too often, as we know well from helping stand up the Security Researcher Legal Defense Fund, individuals working to advance cybersecurity for the public good end up facing criminal charges. The Cybercrime Treaty should not criminalize the work of legitimate cybersecurity researchers and penetration testers, which is designed to protect individual systems and the web as a whole.
UN Member States have an opportunity to strengthen global cybersecurity by adopting a treaty that encourages the criminalization of the most egregious and systemic activities — on which all parties can agree — while adopting a framework for sharing digital evidence that is transparent, grounded in the rule of law, based on pre-existing international frameworks like the Universal Declaration on Human Rights, and aligned with principles of necessity and proportionality. At the same time, Member States should avoid attempts to criminalize activities that raise significant freedom of expression issues, or that actually undercut the treaty’s goal of reducing cybercrime. That will require strengthening critical guardrails and protections.
We urge Member States to heed calls from civil society groups to address critical gaps in the Treaty and revise the text to protect users and security professionals — not endanger the security of the web.
The AI world moves fast, so we’ve been hard at work keeping security apace with recent advancements. One of our approaches, in alignment with Google’s Secure AI Framework (SAIF), is using AI itself to automate and streamline routine and manual security tasks, including fixing security bugs. Last year we wrote about our experiences using LLMs to expand vulnerability testing coverage, and we’re excited to share some updates.
Today, we’re releasing our fuzzing framework as a free, open source resource that researchers and developers can use to improve fuzzing’s bug-finding abilities. We’ll also show you how we’re using AI to speed up the bug patching process. By sharing these experiences, we hope to spark new ideas and drive innovation for a stronger ecosystem security.
Last August, we announced our framework to automate manual aspects of fuzz testing (“fuzzing”) that often hindered open source maintainers from fuzzing their projects effectively. We used LLMs to write project-specific code to boost fuzzing coverage and find more vulnerabilities. Our initial results on a subset of projects in our free OSS-Fuzz service were very promising, with code coverage increased by 30% in one example. Since then, we’ve expanded our experiments to more than 300 OSS-Fuzz C/C++ projects, resulting in significant coverage gains across many of the project codebases. We’ve also improved our prompt generation and build pipelines, which has increased code line coverage by up to 29% in 160 projects.
How does that translate to tangible security improvements? So far, the expanded fuzzing coverage offered by LLM-generated improvements allowed OSS-Fuzz to discover two new vulnerabilities in cJSON and libplist, two widely used projects that had already been fuzzed for years. As always, we reported the vulnerabilities to the project maintainers for patching. Without the completely LLM-generated code, these two vulnerabilities could have remained undiscovered and unfixed indefinitely.
Fuzzing is fantastic for finding bugs, but for security to improve, those bugs also need to be patched. It’s long been an industry-wide struggle to find the engineering hours needed to patch open bugs at the pace that they are uncovered, and triaging and fixing bugs is a significant manual toll on project maintainers. With continued improvements in using LLMs to find more bugs, we need to keep pace in creating similarly automated solutions to help fix those bugs. We recently announced an experiment doing exactly that: building an automated pipeline that intakes vulnerabilities (such as those caught by fuzzing), and prompts LLMs to generate fixes and test them before selecting the best for human review.
This AI-powered patching approach resolved 15% of the targeted bugs, leading to significant time savings for engineers. The potential of this technology should apply to most or all categories throughout the software development process. We’re optimistic that this research marks a promising step towards harnessing AI to help ensure more secure and reliable software.
Since we’ve now open sourced our framework to automate manual aspects of fuzzing, any researcher or developer can experiment with their own prompts to test the effectiveness of fuzz targets generated by LLMs (including Google’s VertexAI or their own fine-tuned models) and measure the results against OSS-Fuzz C/C++ projects. We also hope to encourage research collaborations and to continue seeing other work inspired by our approach, such as Rust fuzz target generation.
If you’re interested in using LLMs to patch bugs, be sure to read our paper on building an AI-powered patching pipeline. You’ll find a summary of our own experiences, some unexpected data about LLM’s abilities to patch different types of bugs, and guidance for building pipelines in your own organizations.