January 4, 2018

More details about mitigations for the CPU Speculative Execution issue

Yesterday, Google’s Project Zero team posted detailed technical information on three variants of a new security issue involving speculative execution on many modern CPUs. Today, we’d like to share some more information about our mitigations and performance.

In response to the vulnerabilities that were discovered we developed a novel mitigation called “Retpoline” -- a binary modification technique that protects against “branch target injection” attacks. We shared Retpoline with our industry partners and have deployed it on Google’s systems, where we have observed negligible impact on performance.

In addition, we have deployed Kernel Page Table Isolation (KPTI) -- a general purpose technique for better protecting sensitive information in memory from other software running on a machine -- to the entire fleet of Google Linux production servers that support all of our products, including Search, Gmail, YouTube, and Google Cloud Platform.

There has been speculation that the deployment of KPTI causes significant performance slowdowns. Performance can vary, as the impact of the KPTI mitigations depends on the rate of system calls made by an application. On most of our workloads, including our cloud infrastructure, we see negligible impact on performance.

In our own testing, we have found that microbenchmarks can show an exaggerated impact. Of course, Google recommends thorough testing in your environment before deployment; we cannot guarantee any particular performance or operational impact.

Speculative Execution and the Three Methods of Attack

In addition, to follow up on yesterday’s post, today we’re providing a summary of speculative execution and how each of the three variants work.

In order to improve performance, many CPUs may choose to speculatively execute instructions based on assumptions that are considered likely to be true. During speculative execution, the processor is verifying these assumptions; if they are valid, then the execution continues. If they are invalid, then the execution is unwound, and the correct execution path can be started based on the actual conditions. It is possible for this speculative execution to have side effects which are not restored when the CPU state is unwound, and can lead to information disclosure.

Project Zero discussed three variants of speculative execution attack. There is no single fix for all three attack variants; each requires protection independently.

  • Variant 1 (CVE-2017-5753), “bounds check bypass.” This vulnerability affects specific sequences within compiled applications, which must be addressed on a per-binary basis.
  • Variant 2 (CVE-2017-5715), “branch target injection”. This variant may either be fixed by a CPU microcode update from the CPU vendor, or by applying a software mitigation technique called “Retpoline” to binaries where concern about information leakage is present. This mitigation may be applied to the operating system kernel, system programs and libraries, and individual software programs, as needed.
  • Variant 3 (CVE-2017-5754), “rogue data cache load.” This may require patching the system’s operating system. For Linux there is a patchset called KPTI (Kernel Page Table Isolation) that helps mitigate Variant 3. Other operating systems may implement similar protections - check with your vendor for specifics.

Variant 1: bounds check bypass (CVE-2017-5753)
This attack variant allows malicious code to circumvent bounds checking features built into most binaries. Even though the bounds checks will still fail, the CPU will speculatively execute instructions after the bounds checks, which can access memory that the code could not normally access. When the CPU determines the bounds check has failed, it discards any work that was done speculatively; however, some changes to the system can be still observed (in particular, changes to the state of the CPU caches). The malicious code can detect these changes and read the data that was speculatively accessed.

The primary ramification of Variant 1 is that it is difficult for a system to run untrusted code within a process and restrict what memory within the process the untrusted code can access.

In the kernel, this has implications for systems such as the extended Berkeley Packet Filter (eBPF) that takes packet filterers from user space code, just-in-time (JIT) compiles the packet filter code, and runs the packet filter within the context of kernel. The JIT compiler uses bounds checking to limit the memory the packet filter can access, however, Variant 1 allows an attacker to use speculation to circumvent these limitations.

Mitigation requires analysis and recompilation so that vulnerable binary code is not emitted. Examples of targets which may require patching include the operating system and applications which execute untrusted code.
Variant 2: branch target injection (CVE-2017-5715)
This attack variant uses the ability of one process to influence the speculative execution behavior of code in another security context (i.e., guest/host mode, CPU ring, or process) running on the same physical CPU core.

Modern processors predict the destination for indirect jumps and calls that a program may take and start speculatively executing code at the predicted location. The tables used to drive prediction are shared between processes running on a physical CPU core, and it is possible for one process to pollute the branch prediction tables to influence the branch prediction of another process or kernel code.

In this way, an attacker can cause speculative execution of any mapped code in another process, in the hypervisor, or in the kernel, and potentially read data from the other protection domain using techniques like Variant 1. This variant is difficult to use, but has great potential power as it crosses arbitrary protection domains.
Mitigating this attack variant requires either installing and enabling a CPU microcode update from the CPU vendor (e.g., Intel's IBRS microcode), or applying a software mitigation (e.g., Google's Retpoline) to the hypervisor, operating system kernel, system programs and libraries, and user applications.
Variant 3: rogue data cache load (CVE-2017-5754)
This attack variant allows a user mode process to access virtual memory as if the process was in kernel mode. On some processors, the speculative execution of code can access memory that is not typically visible to the current execution mode of the processor; i.e., a user mode program may speculatively access memory as if it were running in kernel mode.

Using the techniques of Variant 1, a process can observe the memory that was accessed speculatively. On most operating systems today, the page table that a process uses includes access to most physical memory on the system, however access to such memory is limited to when the process is running in kernel mode. Variant 3 enables access to such memory even in user mode, violating the protections of the hardware.
Mitigating this attack variant requires patching the operating system. For Linux, the patchset that mitigates Variant 3 is called Kernel Page Table Isolation (KPTI). Other operating systems/providers should implement similar mitigations.

Mitigations for Google products

You can learn more about mitigations that have been applied to Google’s infrastructure, products, and services here.