Google Online Security Blog: December 2021

Apache Log4j Vulnerability

December 17, 2021

latest versionCVE-2021-44228CVE-2021-45046AndroidChrome OS Chrome BrowserGoogle Cloudspecific advisoryGoogle Marketing Platform,including Google Ads YouTube

Understanding the Impact of Apache Log4j Vulnerability

December 17, 2021

Posted by James Wetter and Nicky Ringland, Open Source Insights Team

Editors Note:
The below numbers were calculated based on both log4j-core and log4j-api, as both were listed on the CVE. Since then, the CVE has been updated with the clarification that only log4j-core is affected.
The ecosystem impact numbers for just log4j-core, as of 19th December are over 17,000 packages affected, which is roughly 4% of the ecosystem. 25% of affected packages have fixed versions available.

The linked list, which continues to be updated, only includes packages which depend on log4j-core.Maven Central repository12

This vulnerability has captivated the information security ecosystem since its disclosure on December 9th because of both its severity and widespread impact. As a popular logging tool, log4j is used by tens of thousands of software packages (known as artifacts in the Java ecosystem) and projects across the software industry. User’s lack of visibility into their dependencies and transitive dependencies has made patching difficult; it has also made it difficult to determine the full blast radius of this vulnerability. Using Open Source Insights, a project to help understand open source dependencies, we surveyed all versions of all artifacts in the Maven Central Repository to determine the scope of the issue in the open source ecosystem of JVM based languages, and to track the ongoing efforts to mitigate the affected packages.

How widespread is the log4j vulnerability?

As of December 16, 2021, we found that 35,863 of the available Java artifacts from Maven Central depend on the affected log4j code. This means that more than 8% of all packages on Maven Central have at least one version that is impacted by this vulnerability. (These numbers do not encompass all Java packages, such as directly distributed binaries, but Maven Central is a strong proxy for the state of the ecosystem.)

As far as ecosystem impact goes, 8% is enormous. The average ecosystem impact of advisories affecting Maven Central is 2%, with the median less than 0.1%.

Direct dependencies account for around 7,000 of the affected artifacts, meaning that any of its versions depend upon an affected version of log4j-core or log4j-api, as described in the CVEs. The majority of affected artifacts come from indirect dependencies (that is, the dependencies of one’s own dependencies), meaning log4j is not explicitly defined as a dependency of the artifact, but gets pulled in as a transitive dependency.

What is the current progress in fixing the open source JVM ecosystem?
We counted an artifact as fixed if the artifact had at least one version affected and has released a greater stable version (according to semantic versioning) that is unaffected. An artifact affected by log4j is considered fixed if it has updated to 2.16.0 or removed its dependency on log4j altogether.

At the time of writing, nearly five thousand of the affected artifacts have been fixed. This represents a rapid response and mammoth effort both by the log4j maintainers and the wider community of open source consumers.

That leaves over 30,000 artifacts affected, many of which are dependent on another artifact to patch (the transitive dependency) and are likely blocked.

Why is fixing the JVM ecosystem hard?

Most artifacts that depend on log4j do so indirectly. The deeper the vulnerability is in a dependency chain, the more steps are required for it to be fixed. The following diagram shows a histogram of how deeply an affected log4j package (core or api) first appears in consumers dependency graphs. For greater than 80% of the packages, the vulnerability is more than one level deep, with a majority affected five levels down (and some as many as nine levels down). These packages will require fixes throughout all parts of the tree, starting from the deepest dependencies first.

Another difficulty is caused by ecosystem-level choices in the dependency resolution algorithm and requirement specification conventions.

In the Java ecosystem, it’s common practice to specify “soft” version requirements — exact versions that are used by the resolution algorithm if no other version of the same package appears earlier in the dependency graph. Propagating a fix often requires explicit action by the maintainers to update the dependency requirements to a patched version.

This practice is in contrast to other ecosystems, such as npm, where it’s common for developers to specify open ranges for dependency requirements. Open ranges allow the resolution algorithm to select the most recently released version that satisfies dependency requirements, thereby pulling in new fixes. Consumers can get a patched version on the next build after the patch is available, which propagates up the dependencies quickly. (This approach is not without its drawbacks; pulling in new fixes can also pull in new problems.)

How long will it take for this vulnerability to be fixed across the entire ecosystem?

It’s hard to say. We looked at all publicly disclosed critical advisories affecting Maven packages to get a sense of how quickly other vulnerabilities have been fully addressed. Less than half (48%) of the artifacts affected by a vulnerability have been fixed, so we might be in for a long wait, likely years.

But things are looking promising on the log4j front. After less than a week, 4,620 affected artifacts (~13%) have been fixed. This, more than any other stat, speaks to the massive effort by open source maintainers, information security teams and consumers across the globe.

Where to focus next?

Thanks and congratulations are due to the open source maintainers and consumers who have already upgraded their versions of log4j. As part of our investigation, we pulled together a list of 500 affected packages with some of the highest transitive usage. If you are a maintainer or user helping with the patching effort, prioritizing these packages could maximize your impact and unblock more of the community.

We encourage the open source community to continue to strengthen security in these packages by enabling automated dependency updates and adding security mitigations. Improvements such as these could qualify for financial rewards from the Secure Open Source Rewards program.

You can explore your package dependencies and their vulnerabilities by using Open Source Insights.

Improving OSS-Fuzz and Jazzer to catch Log4Shell

December 16, 2021

Posted by Jonathan Metzman, Google Open Source Security Team
The discovery of the Log4Shell vulnerability has set the internet on fire. Similar to shellshock and heartbleed, Log4Shell is just the latest catastrophic vulnerability in software that runs the internet. Our mission as the Google Open Source Security Team is to secure the open source libraries the world depends on, such as Log4j. One of our capabilities in this space is OSS-Fuzz, a free fuzzing service that is used by over 500 critical open source projects and has found more than 7,000 vulnerabilities in its lifetime.

We want to empower open source developers to secure their code on their own. Over the next year we will work on better automated detection of non-memory corruption vulnerabilities such as Log4Shell. We have started this work by partnering with the security company Code Intelligence to provide continuous fuzzing for Log4j, as part of OSS-Fuzz. Also as part of this partnership, Code-Intelligence improved their Jazzer fuzzing engine to make it capable of detecting remote JNDI lookups. We have awarded Code Intelligence $25,000 for this effort and will continue to work with them on securing the open source ecosystem.

Caption: OSS-Fuzz and Jazzer finding the Log4Shell Vulnerability

Vulnerabilities like Log4Shell are an eye-opener for the industry in terms of new attack vectors. With OSS-Fuzz and Jazzer, we can now detect this class of vulnerability so that they can be fixed before they become a problem in production code.

Over the past year we have made a number of investments to strengthen the security of critical open source projects, and recently announced our $10 billion commitment to cybersecurity defense including $100 million to support third-party foundations that manage open source security priorities and help fix vulnerabilities.

We appreciate the maintainers, security engineers and incident responders that are working to mitigate Log4j and make our internet ecosystem safer.

Check out our documentation to get started using OSS-Fuzz.

Empowering the next generation of Android Application Security Researchers

December 14, 2021

Posted by Jon Bottarini, Security Program Manager & Lena Katib, Strategic Partnerships Manager

The external security researcher community plays an integral role in making the Google Play ecosystem safe and secure. Through this partnership with the community, Google has been able to collaborate with third-party developers to fix thousands of security issues in Android applications before they are exploited and reward security researchers for their hard work and dedication. In order to empower the next generation of Android security researchers, Google has collaborated with industry partners including HackerOne and PayPal to host a number of Android App Hacking Workshops. These workshops are an effort designed to educate security researchers and cybersecurity students of all skill levels on how to find Android application vulnerabilities through a series of hands-on working sessions, both in-person and virtual. Through these workshops, we’ve seen attendees from groups such as Merritt College's cybersecurity program and alumni of Hack the Hood go on to report real-world security vulnerabilities to the Google Play Security Rewards program. This reward program is designed to identify and mitigate vulnerabilities in apps on Google Play, and keep Android users, developers and the Google Play ecosystem safe. Today, we are releasing our slide deck and workshop materials, including source code for a custom-built Android application that allows you to test your Android application security skills in a variety of capture the flag style challenges. These materials cover a wide range of techniques for finding vulnerabilities in Android applications. Whether you’re just getting started or have already found many bugs - chances are you’ll learn something new from these challenges! If you get stuck and need a hint on solving a challenge, the solutions for each are available in the Android App Hacking Workshop here. As you work through the challenges and learn more about the techniques and tips described in our workshop materials, we’d love to hear your feedback. Additional Resources:

If you want to learn more about how to prepare, launch, and run a Vulnerability Disclosure Program (VDP) or discover how to work with external security researchers, check out our VDP course here.

If you’re a developer looking to build more secure applications, check out Android app security best practices here.

Exploring Container Security: A Storage Vulnerability Deep Dive

December 2, 2021

Posted by Fabricio Voznika and Mauricio Poppe, Google Cloud high severity vulnerabilityvulnerability management in open-source Kubernetesthe Kubernetes Security Response CommitteeKubernetes Filesystems: Intro to Volume SubpathCVE-2021-25741, inside a container, andvolume subpathA previous fixtime of check to time of use (TOCTOU)A New TOCTOU Vulnerability: CVE-2021-25741renameat(2)The expectation and the attack outcomemount(8)The fix is in
Once the problem was well understood, we fixed it inside Kubernetes and quickly released the fix to GKE and Anthos. If GKE auto-upgrade is enabled in your clusters there's no action on your part for this vulnerability, your nodes have already been patched. We strongly recommend that customers utilize auto-upgrades. Auto-upgrade gives peace of mind that your clusters are running with the latest patches.

GKE released a Google Kubernetes Engine security bulletin on this vulnerability, which detailed what customers can do to immediately remediate this issue across GKE and Anthos. We also provided guidance to customers who manually manage their node versions, ensuring that fixed releases were available in every region for our Static and Release Channels.

Moving forward
Google continues to invest heavily in the security of GKE and Kubernetes. We encourage users interested in finding vulnerabilities to participate in the Kubernetes bug bounty program and in the Google Vulnerability Rewards Program (VRP) which was recently expanded to cover GKE vulnerabilities. For the latest guidance on security issues, please follow our GKE Security Bulletins.

Security Blog

Apache Log4j Vulnerability

Understanding the Impact of Apache Log4j Vulnerability

Improving OSS-Fuzz and Jazzer to catch Log4Shell

Empowering the next generation of Android Application Security Researchers

Exploring Container Security: A Storage Vulnerability Deep Dive

Labels

Archive

Feed