Spectre & Meltdown in the Cloud

Your guide to best practices for patching Speculative Execution Vulnerabilities no matter which stack you’re running.

Published in

CloudCover

6 min readJan 7, 2018

https://pixabay.com/en/halloween-pumpkin-gourd-creepy-2870607/

This post will be updated as and when CloudCover finds any new information around Speculative Execution Vulnerabilities.

Patching linux-based cloud operating systems

This section deals with the patches and configuration steps necessary to fix popular linux-based cloud operating environments. These commands and patches should work on any cloud (or datacenter for that matter).

Amazon Linux

An updated kernel for Amazon Linux is available within the Amazon Linux repositories. Instances launched with the default Amazon Linux configuration on or after 10:45 PM (GMT) January 3rd, 2018 will automatically include the updated package. Customers with existing Amazon Linux AMI instances should run the following command to ensure they receive the updated package:

yum update kernel

More information on this bulletin is available at the Amazon Linux AMI Security Center

Ubuntu

As of writing, Ubuntu HAVE NOT released a kernel patch for any release of Ubuntu. We highly recommend tracking this link closely

Debian

As of this email, Debian has released a patch for stretch and wheezy, jessie and buster are still vulnerable.

CentOS/RHEL/Fedora/Oracle/Scientific Linux

Use the following yum command to patch systems:

sudo yum update

Fedora Linux

Use the following command:

sudo dnf — refresh update kernel

sudo dnf update

Patching Windows Server

Customers should take the following actions to help protect against the vulnerabilities:

Apply the Windows operating system update. For details on how to enable this update, see Microsoft Knowledge Base Article 4072699.
Make necessary configuration changes to enable protection.
Apply an applicable firmware update from the OEM device manufacturer.

Important: Customers who only install the Windows update will not receive the benefit of all known protections.

Windows Servers-based machines (physical or virtual) should get the Windows security updates that were released on January 3, 2018, and are available from Windows Update. The following updates are available:

Windows Server, version 1709 (Server Core Installation)
Windows Server 2016
Windows Server 2012 R2
Windows Server 2012 — NOT AVAILABLE
Windows Server 2008 R2
Windows Server 2008 — NOT AVAILABLE

Note: In addition to installing the January security update, a processor microcode update is required. This should be available through your OEM.

Enabling protections on server

Important: This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, view this article in the Microsoft Knowledge Base.

Customers need to enable mitigations to help protect against speculative execution side-channel vulnerabilities.

Enabling these mitigations may affect performance. The actual performance impact will depend on multiple factors, such as the specific chipset in your physical host and the workloads that are running. Microsoft recommends that customers assess the performance impact for their environment and make necessary adjustments.

Your server is at increased risk if it is in one of the following categories:

Hyper-V hosts
Remote Desktop Services Hosts (RDSH)

For physical hosts or virtual machines that are running untrusted code such as containers or untrusted extensions for database, untrusted web content or workloads that run code that is provided from external sources.

Use these registry keys to enable the mitigations on server:

To enable the mitigations:

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverride /t REG_DWORD /d 0 /freg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

To disable the mitigations:

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverride /t REG_DWORD /d 3 /freg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

Verifying protections are enabled

To help confirm whether protections have been enabled, Microsoft has published a PowerShell script that customers can run on their systems. Install and run the script by running the following commands:

PowerShell verification

Install the PowerShell module

PS > Install-Module SpeculationControl

Run the PowerShell module to validate protections are enabled

PS > Get-SpeculationControlSettings

The output of this PowerShell script will look like the following. Enabled protections will show in the output as “True”.

PS C:\> Get-SpeculationControlSettings
Speculation control settings for CVE-2017–5715 [branch target injection]
Hardware support for branch target injection mitigation is present: True
Windows OS support for branch target injection mitigation is present: True
Windows OS support for branch target injection mitigation is enabled: True
Speculation control settings for CVE-2017–5754 [rogue data cache load]
Hardware requires kernel VA shadowing: True
Windows OS support for kernel VA shadow is present: True
Windows OS support for kernel VA shadow is enabled: True
Windows OS support for PCID optimization is enabled: True

Google Cloud Platform

Google Cloud Dataflow

Cloud Dataflow customers must update any streaming pipelines that are launched before 2018–01–05 and are currently running, and restart any batch pipelines that are launched before 2018–01–05. Pipelines launched after 2018–01–05 will be protected.

In the case where updating the streaming pipelines is not possible, Cloud Dataflow customers must drain the pipelines and restart them.

Google Cloud Datalab

Customers with CPU instances older than 12/08/2017 should consider recreating their instances to automatically pick up the newer patched images.

Customers with GPU instances (which use Ubuntu 16.04) should read the Security Bulletins page for more information on OS provider patch status, and patched image versions.

Google Cloud Dataproc

Cloud Dataproc customers who run multiple, untrusted workloads on the same Cloud Dataproc cluster should update these shared clusters to patched images as they become available.

Customers who deploy ephemeral Dataproc clusters on-demand using the default latest image or specifying a <major>.<minor> image version, new cluster deployments will automatically use the newest patched images as soon as they become available, and no customer action is needed

Customers who have long-lived Dataproc clusters or pin to a specific <major>.<minor>.<patch>version number, should subscribe to Dataproc release notes to receive ongoing information about patches as they become available, possibly over the course of multiple patch versions. Customers should then unpin and/or redeploy to use the latest patch versions as soon as they become available.

Google Kubernetes Engine

Kubernetes Engine customers must update their runtime environments so that applications within each runtime environment are protected from each other.

Google Kubernetes Engine customers who use Container-Optimized OS image, and who have autoupgrade enabled, will be updated to patched versions of our COS image as they become available. The COS images for 1.6.13-gke.1, 1.7.11-gke.1, 1.8.4-gke.1, and newer, have been patched for variants 1 and 3. Variant 2 patches are in development and are expected to be released in early March.

If you do not have autoupgrade enabled, you must manually upgrade instead.

Amazon Web Services

Relational Database Service

For RDS PostgreSQL and Aurora PostgreSQL, DB Instances running in the default configuration currently have no customer actions required. AWS will provide the appropriate patches for users of plv8 extensions once they are made available. In the meantime, customers who have enabled plv8 extensions (disabled by default) should consider disabling them and review V8’s guidance at https://github.com/v8/v8/wiki/Untrusted-code-mitigations.

For RDS for SQL Server Database Instances, AWS will release OS and database engine patches as Microsoft makes each available, allowing customers to upgrade at a time of their choosing. Customers who have enabled CLR (disabled by default) should review Microsoft’s guidance on disabling the CLR extension at https://support.microsoft.com/en-us/help/4073225/guidance-for-sql-server

WorkSpaces

AWS will apply security updates released by Microsoft to most AWS WorkSpaces over the coming weekend. Customers should expect their WorkSpaces to reboot during this period.

Important: Bring Your Own License (BYOL) customers, and customers who have changed the default update setting in their WorkSpaces should manually apply the security updates provided by Microsoft.

Updated WorkSpaces bundles will be available with the security updates soon. Customers who have created Custom Bundles should update their bundles to include the security updates themselves. Any new WorkSpaces launched from bundles that do not have the updates will receive patches soon after launch, unless customers have changed the default update setting in their WorkSpaces, in which case they should follow the above steps to manually apply the security updates provided by Microsoft.

WorkSpaces Application Manager (WAM)

We recommend that customers choose one of the following courses of action:

Option 1: Manually apply the Microsoft patches on running instances of WAM Packager and Validator by following the steps provided by Microsoft at https://support.microsoft.com/en-us/help/4072698/windows-server-guidance-to-protect-against-the-speculative-execution. This page provides further instructions and downloads for Windows Server.

Option 2: Rebuild new WAM Packager and Validator EC2 instances from updated AMIs for WAM Packager and Validator which will be available by end of day (2018/01/04).