Guided in-process fuzzing of Chrome components

Posted by Max Moroz, Chrome Security Engineer and Kostya Serebryany, Sanitizer Tsar

In the past, we’ve posted about innovations in fuzzing, a software testing technique used to discover coding errors and security vulnerabilities. The topics have included AddressSanitizer, ClusterFuzz, SyzyASAN, ThreadSanitizer and others.

Today we'd like to talk about libFuzzer (part of the LLVM project), an engine for in-process, coverage-guided, white-box fuzzing:

• By in-process, we mean that we don’t launch a new process for every test case, and that we mutate inputs directly in memory.
• By coverage-guided, we mean that we measure code coverage for every input, and accumulate test cases that increase overall coverage.
• By white-box, we mean that we use compile-time instrumentation of the source code.

LibFuzzer makes it possible to fuzz individual components of Chrome. This means you don’t need to generate an HTML page or network payload and launch the whole browser, which adds overhead and flakiness to testing. Instead, you can fuzz any function or internal API directly. Based on our experience, libFuzzer-based fuzzing is extremely efficient, more reliable, and usually thousands of times faster than traditional out-of-process fuzzing.

Our goal is to have fuzz testing for every component of Chrome where fuzzing is applicable, and we hope all Chromium developers and external security researchers will contribute to this effort.

How to write a fuzz target

With libFuzzer, you need to write only one function, which we call a target function or a fuzz target. It accepts a data buffer and length as input and then feeds it into the code we want to test. And... that’s it!

The fuzz targets are not specific to libFuzzer. Currently, we also run them with AFL, and we expect to use other fuzzing engines in the future.
Sample Fuzzer

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
 std::string buf;
 woff2::WOFF2StringOut out(&buf);
 out.SetMaxSize(30 * 1024 * 1024);
 woff2::ConvertWOFF2ToTTF(data, size, &out);
 return 0;
}
See also the build rule.
Sample Bug

==9896==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62e000022836 at pc 0x000000499c51 bp 0x7fffa0dc1450 sp 0x7fffa0dc0c00

WRITE of size 41994 at 0x62e000022836 thread T0

SCARINESS: 45 (multi-byte-write-heap-buffer-overflow)

   #0 0x499c50 in __asan_memcpy

   #1 0x4e6b50 in Read third_party/woff2/src/buffer.h:86:7

   #2 0x4e6b50 in ReconstructGlyf third_party/woff2/src/woff2_dec.cc:500

   #3 0x4e6b50 in ReconstructFont third_party/woff2/src/woff2_dec.cc:917

   #4 0x4e6b50 in woff2::ConvertWOFF2ToTTF(unsigned char const*, unsigned long, woff2::WOFF2Out*) third_party/woff2/src/woff2_dec.cc:1282

   #5 0x4dbfd6 in LLVMFuzzerTestOneInput testing/libfuzzer/fuzzers/convert_woff2ttf_fuzzer.cc:15:3

Check out our documentation for additional information.

Integrating LibFuzzer with ClusterFuzz

ClusterFuzz is Chromium’s infrastructure for large scale fuzzing. It automates crash detection, report deduplication, test minimization, and other tasks. Once you commit a fuzz target into the Chromium codebase (examples), ClusterFuzz will automatically pick it up and fuzz it with libFuzzer and AFL. 

ClusterFuzz supports most of the libFuzzer features like dictionaries, seed corpus and custom options for different fuzzers. Check out our Efficient Fuzzer Guide to learn how to use them.

Besides the initial seed corpus, we store, minimize, and synchronize the corpora for every fuzzer and across all bots. This allows us to continuously increase code coverage over time and find interesting bugs along the way.

ClusterFuzz uses the following memory debugging tools with libFuzzer-based fuzzers:

• AddressSanitizer (ASan): 500 GCE VMs
• MemorySanitizer (MSan): 100 GCE VMs
• UndefinedBehaviorSanitizer (UBSan): 100 GCE VMs

Sample Fuzzer Statistics

It’s important to track and analyze performance of fuzzers. So, we have this dashboard to track fuzzer statistics, that is accessible to all chromium developers:

Overall statistics for the last 30 days:

• 120 fuzzers
• 112 bugs filed
• Aaaaaand…. 14,366,371,459,772 unique test inputs!

Analysis of the bugs found so far

Looking at the 324 bugs found so far, we can say that ASan and MSan have been very effective memory tools for finding security vulnerabilities. They give us comparable numbers of crashes, though ASan crashes usually are more severe than MSan ones. LSan (part of ASan) and UBSan have a great impact for Stability - another one of our 4 core principles.

Extending Chrome’s Vulnerability Reward Program

Under Chrome's Trusted Researcher Program, we invite submission of fuzzers. We run them for you on ClusterFuzz and automatically nominate bugs they find for reward payments.

Today we're pleased to announce that the invite-only Trusted Researcher Program is being replaced with the Chrome Fuzzer Program which encourages fuzzer submissions from all, and also covers libFuzzer-based fuzzers! Full guidelines are listed on Chrome’s Vulnerability Reward Program page.