How to record system operator activities on AWS using Amazon AppStream 2.0 and Session Manager

16 min readAug 11, 2022

Back in 2020, when I was still at AWS, I wrote a blog post about how to record a video of streaming sessions in Amazon AppStream 2.0, a fully managed service for remotely accessing non-persistent desktops and applications.

Since then, AWS added support in AppStream 2.0 for Amazon Linux 2, a cost-effective alternative to Windows Server, and Elastic Fleets, a serverless fleet type that eliminates the need of managing streaming instances. AWS also released several enhancements for Session Manager, a feature of AWS Systems Manager that lets you connect to EC2 instances, such as port forwarding to remote hosts. So, I thought it would be useful to update the original blog post and take advantage of the new features.

Moreover, this time I will apply it to a more global use case of recording the activities of system operators in an AWS environment. Beyond being a good security practice, several compliance frameworks require to log the activities of administrators, or more generally of any system operators, such as for example HDS — a certification needed in France to host healthcare data — see the control 4.4.6.10. As I found little documentation on how to implement this, I hope this article can help those with similar needs.

Objectives of the solution

The solution proposed in this article aims to meet several needs:

1/ The first objective, as stated in the title, is to record human activities on the systems to protect. Since many applications rely on graphical interfaces, this consists of capturing a video of the screen that those systems expose to the system operators. This article covers the following types of activities:

Operations on EC2 instances, such as SSH or RDP interactive sessions;
Operations on “middlewares” managed by AWS without OS access, such as querying a RDS MySQL database from a SQL client, or accessing a OpenSearch Dashboard from a browser;
Operations on the AWS management console and APIs. You might ask, “Why recording AWS activities when CloudTrail exists?” CloudTrail logs are comprehensive but complex to investigate. Moreover, some actions cannot be recorded by CloudTrail, such as writing data (s3:PutObject) to an unmanaged S3 bucket. That is why a video capture of the actions made may be complementary, if not easier to analyze.

Note that, for critical systems or systems containing sensitive data, it is best to avoid human access as much as possible. Instead, changes should be deployed using a CI/CD pipeline, and actions should be logged at the pipeline level. However, in many situations, it is unrealistic to remove all human access, whether for troubleshooting, ad-hoc changes, or manually configured applications… Hence the need for logging system operator activities.

2/ The second objective is to reduce the risk of data exfiltration. In other words, we want to prevent sensitive data from being downloaded and then shared and losing control of it.

In this article, AppStream 2.0 is used as the “entry point” to the systems to record, and prevents the transfer of data with the outside world apart from the screen display of the streaming sessions. Other possible measures will be detailed further in this article.

However, it is extremely difficult to prevent any possible data exfiltration, that is why I talk about reducing the risk. For example, it is hard to prevent a system operator from downloading sensitive data from a critical AWS account and then uploading that same data to an untrusted AWS account. However, the proposed solution at least records this action, and perhaps discourages violators if they know they are being recorded.

3/ The last objective is to capture the reason why system operators had to connect to protected systems, or even obtain prior validation, and associate the reason with the corresponding records. This should also discourage human access without a good reason.

Traditional approach to record human activities

The traditional approach to record operations made on servers is to deploy a bastion host — either open source (Apache Guacamole…) or commercial (Wallix, PrivX, CyberArk…) — which consists of a hardened jump server whose sole purpose is to serve as a trusted relay for inbound connections to internal resources.

While each existing solution has its own specificities, private connectivity is generally required between the bastion host and the internal resources (i.e. traffic can be routed and not blocked by firewall rules). System operators are authenticated either at the bastion level and/or at the internal resource level. For example, users authenticate at the resource level when internal Windows servers are integrated with Active Directory. On the other hand, the bastion host can authenticate users before establishing SSH sessions, then dynamically and transparently rotate the key pair in the target server.

Activity recording takes place at the bastion level, provided that the bastion is able to “decode” the flow between the bastion user and the internal resource, and not just transmit unintelligible bits, as with SSH ProxyCommand or port forwarding.

The proposed solution

How the solution works is described below and is illustrated in Figure 2. Later in this post, in the “Implementation” section, you can find high-level instructions about how to implement the solution.

Key components of the solution

An Elastic fleet configured to use Amazon Linux 2 streaming instances. Elastic Fleets remove the need of maintaining and right-scaling streaming instances. Amazon Linux 2 is more cost-effective than Windows Server, and doesn’t require to pay a license fee per user (RDS SAL).
Multiple applications or tools made available in the streaming instances. They are either installed by a session script at session launch, or associated as applications to Elastic Fleets. At least, the following applications are used in this article: the Session Manager plugin for AWS CLI, a web browser like Google Chrome, a RDP client such as Remmina, and a SQL client like DbVisualizer.
Note that AppStream 2.0 users (system operators) don’t have root access to streaming instances to install their own applications. Therefore, you may need to install additional applications, if no “portable” version exists.
A dedicated VPC in an AWS account dedicated to the solution, in which streaming instances are created. The VPC uses NAT Gateways to provide Internet connectivity to the streaming instances.
A S3 bucket to store the recorded video files, and other associated session metadata.
EC2 instances to which system operators may want to connect. These instances must have the SSM agent installed and an EC2 role attached that allows to use Session Manager, a feature of AWS Systems Manager (the AWS-managed policy AmazonSSMManagedInstanceCore is sufficient). The SSM agent is preinstalled on some AMIs provided by AWS.
Note that you can create a Config Rule with remediation enabled to attach a pre-existing EC2 role with the minimum required permissions for Session Manager if no EC2 role is attached to an instance, or to add the minimum required policies to the attached EC2 role if permissions are missing.

How system operators use the solution

**Figure 2**: Architecture of the solution

System operators initiate a streaming session to one of the streaming instances, and interact with the streaming instance’s GNOME desktop via their web browser.
The streaming instance is configured to execute a session script at session launch to record and upload to S3 a video of the desktop screen using FFmpeg, a popular media framework. This script works the same way as in the initial blog post, but has been adapted for Linux. The script runs as root and therefore cannot be interrupted by system operators.
System operators open a terminal and configure the AWS credentials to use by the AWS CLI in the streaming instance. For example, they can run the command aws configure and enter their secret key and access key, or they can use the web browser available in the streaming instance to authenticate to AWS SSO, retrieve short-term credentials and set AWS environment variables. The IAM user or role matching the AWS credentials must be in the same AWS account as the EC2 instance to connect to, and must be authorized to create Session Manager sessions (ssm:StartSession).
To connect to a Linux EC2 instance using SSH: Systems operate run the command aws ssm create-session --target instance_id in the AWS CLI (with instance_id the ID of the target instance) and a SSH terminal to the EC2 instance opens on the streaming instance’s desktop — see Session Manager documentation.
To connect to a Windows EC2 instance using RDP: System operators run the command aws ssm create-session --target instance_id --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["3389"], "localPortNumber":["53389"]}' in the AWS CLI to forward the RDP port on the Windows instance (3389) to a local port on the streaming instance (53389) — see Session Manager documentation. Then, system operators use the RDP client Remmina available in the streaming instance to establish a connection to the local TCP port 53389, and remotely access the Windows instance screen.
To connect to a “managed” middleware, such as a MySQL RDS instance: System operators run the command aws ssm create-session --target instance_id --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters '{"host":["mydb.region.rds.amazonaws.com"], "portNumber":["3306"], "localPortNumber":["3306"]}' in the AWS CLI to forward the MySQL port of the MySQL RDS instance (3306) to a local port on the streaming instance (53306) via the EC2 instance instance_id— see Session Manager documentation. Then, system operators use the SQL client DbVisualizer available in the streaming instance to connect to the database.
To connect to the AWS management console: System operators use the web browser Google Chrome available in the streaming instance. They can also use the AWS CLI in the terminal to run programmatic commands, using the same credentials configured in Step 3.

Benefits of the solution

Private connectivity is not needed between streaming instances and EC2 instances, as long as both can access the public Session Manager APIs. Therefore, it avoids exposing all possible EC2 instances to the streaming instances, it simplifies IP addressing, and it makes it possible to have “Application” VPCs with overlapping CIDR ranges.
The use of Session Manager to connect to EC2 instances removes the need to open inbound ports, such as TCP 22 for SSH, because it is the SSM agent that establishes an outbound connection to Session Manager.
The use of Session Manager enables to control access to EC2 instances using AWS IAM, and removes the need for SSH keys. Therefore, if you revoke a system operator’s access to AWS, his or her access to EC2 instances is automatically revoked.

Sample measures to prevent system operators from bypassing the recording

1/ System operators should not be able to bypass the streaming instances and connect directly to EC2 instances, or to any other resources to protect. For example, if the human activities on a RDS MySQL database should be recorded, the TCP port 3306 should only be accessible from or via a streaming instance.

One possible solution is to restrict inbound traffic using Network ACLs (NACLs) and prohibit the modification of NACLs by system operators. Another solution is to automatically remove any security group inbound rules that exposes certain ports (SSH, RDP, MySQL…) to non-recorded sources, using a custom Config Rule with remediation enabled.

2/ System operators should not be able to connect to recorded AWS accounts via the management console or APIs, outside of the streaming instance. One possible solution is to deny any operations not originating from the streaming instance, via system operators’ IAM role or user in recorded AWS accounts. This can be implemented using the IAM condition key aws:SourceIp and the public IP address of the NAT Gateways.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Deny",
        "Action": "*",
        "Resource": "*",
        "Condition": {
            "NotIpAddress": {
                "aws:SourceIp": [ "NAT Gateway public IPs" ]
            },
            "Bool": {
                "aws:ViaAWSService": "false"
            }
        }
    }
}

3/ System operators should not be able to create an IAM role in a recorded AWS account, and assume it from a non-recorded AWS account, such as a development account.

One possible solution is to create a SCP similar to the policy above to restrict operations for all IAM principals in a given AWS account. However, it is unclear if this SCP could impact your applications, as the AWS documentation doesn’t specify how the condition key aws:ViaAWSService is evaluated with roles used by AWS services, such as EC2 roles.

Another solution is to automatically remove untrusted IAM principals from the trust policy of IAM roles, using a custom Config rule with remediation enabled. For example, you may want to allow only AWS accounts dedicated to security controls and CI/CD pipelines to assume IAM roles in recorded AWS accounts, without source IP restriction.

4/ Obviously, system operators should not be able to disable the solution or modify its resources. That is why it is recommended to use a dedicated AWS account. However, note that those who build and maintain the solution cannot be recorded (dilemma of the Chicken or the Egg).

Sample measures to reduce the risk of data exfiltration

1/ System operators should not be able to copy data or download files from streaming instances to their local workstation. For example, you don’t want them to download an export of a sensitive database. One possible solution is to disable copy/paste from streaming instances, file download and printing (see AppStream 2.0 documentation).

2/ System operators should not be able to export sensitive data to an untrusted remote host or an untrusted AWS account, such as sending a file to an untrusted S3 bucket.

This is very complex to enforce, and requires the use of an outbound proxy with a whitelist of remote hosts, instead of NAT Gateways. You can use AWS Network Firewall as a managed service, or an open source software such as Squid (see this AWS whitepaper about centralized egress).

A simpler measure, although not sufficient, consists of preventing system operators from assuming an IAM role or sending data to an untrusted AWS account using the IAM condition key aws:ResourceAccount (or aws:ResourceOrgPaths). The following policy can be enforced as a SCP, or added to the IAM users or roles that system operators may use.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Deny",
        "Action": [
            "sts:AssumeRole",
            "s3:PutObject",
            ...
        ],
        "Resource": "*",
        "Condition": {
            "StringNotEquals": {
                "aws:ResourceAccount": "this_account_id"
            }
        }
    }
}

3/ System operators should not be able to download sensitive data from a protected AWS account to the streaming instance’s local disk, and then sign in to an untrusted AWS account and upload that data from the streaming instance.

This is also very complex to enforce. This cannot be done using an outbound proxy with a hostname whitelist, as the hostnames are the same for both trusted and untrusted AWS accounts.

One possible solution is to create VPC endpoints in the streaming instances’ VPC, and configure endpoint policies (when supported) to deny requests from streaming instances on resources that reside in untrusted AWS accounts, using the same condition keys as above.

Note: Although the measures 2/ and 3/ are very complex to enforce, and this goes beyond the scope of this article, any violations observable on the screen of the streaming instance would still be recorded by the solution.

Evolving the architecture to capture the reason for access

AppStream 2.0 supports multiple methods of authentication (built-in user directory, SAML…) but none of them natively allow system operators to enter a “reason” before the streaming session starts.

The solution that I propose to capture the reason is illustrated in Figure 3, with AWS SSO as the example of identity provider:

**Figure 3**: Architecture add-on to capture the reason for access

Key components:

An existing identity provider to authenticate system operators, such as AWS SSO, Okta or Azure AD.
An Application Load Balancer (ALB) configured to authenticate users before requests are forwarded to the target group (see ALB documentation). Note that ALB only supports OpenID authentication. Therefore, if your identity provider only supports SAML, like AWS SSO, you can use a Cognito User Pool with a SAML external identity provider to “convert” SAML to OpenID.
An application running behind the ALB. For example, it can be an Elastic Beanstalk platform, or a serverless deployment in AWS Lambda (ALB can route requests to AWS Lambda directly). In my case, a simple Flask application does the job. The service role attached to the EC2 instances or the Lambda function must have permissions to call appstream:CreateStreamingURL.
A persistent storage layer such as a DynamoDB table to log the captured reasons, and any other data generated during the access request process.

Flow steps:

System operators enter the URL of the ALB in their web browser (e.g. bastion-aws.mycompany.com).
The ALB redirects system operators to the identity provider. Once authenticated, the identity provider sends them back to the ALB
The ALB forwards the request to the application. The application retrieves the user identity, such as system operators’ username, from the HTTP headers X-AMZN-OIDC-* passed by the ALB.

What happens next depends on the business process you wish to implement. For example, you can display a message indicating that system operator activities are recorded and explain the rules to follow. System operators are then invited to enter a reason. After submitting the form, the application generates a pre-signed AppStream 2.0 streaming URL and redirects system operators to that URL. By storing the reason, the username and the request time in the persistent database, you can easily match the recorded video files with a person and a reason.

You can also implement a more complex workflow, either by building the application from scratch or by leveraging an existing ITSM tool, such as ServiceNow. For example, the process may require the validation of a manager or a peer before the streaming URL is generated.

Implementation

This section gives high-level instructions about how to implement the solution. It is a not a prescriptive and comprehensive guide, and does not cover the sample measures sections above, nor the architecture to capture reasons for access.

Step 1: Create a new S3 bucket

Create a new S3 bucket in the AWS account and the region where you want to deploy the solution.
Create two empty folders in the bucket: apps to store session scripts and Apps files, and records to store the solution records.
Configure the bucket policy to allow AppStream 2.0 to read content in the apps folder (see documentation).

Step 2: Prepare the session scripts

Download the content of this Github repository. Create an empty folder in your local workstation, and move all the .sh and .json files of the repository to this folder.
Download the Linux AMD64 static build for the latest FFmpeg release — the official website currently redirects to this URL. Extract the archive and copy the executable ffmpeg to the folder that contains the .sh and .json files.
Edit the file variables.sh to specify the name of the S3 bucket where solution records will be stored, the prefix to use (records/) and the bucket region (e.g. eu-west-1).
Create a ZIP archive session-scripts.zip that contains the session scripts and the FFmpeg executable. Note that the files must be at the “root” of the ZIP archive, and not be contained in a folder.
Upload the ZIP archive to a S3 bucket, in the apps folder.

Step 3: Create the App Blocks and Applications in AppStream 2.0

Follow the instructions in this blog post to package the RDP client Remmina, and to expose the built-in SSH client (Terminal) as an App.
Follow the instructions in this blog post to package the web browser Google Chrome.
Repeat similar instructions for all applications that you want to make available in the streaming instances.

At the end of this step, for each application, you must have uploaded a VHD file, a mount script and an icon to the folder apps of the S3 bucket, and you must have created an App Block and an Application in AppStream 2.0.

Note that the Session Manager plugin for the AWS CLI is not associated as an App, but is installed by the session script at launch, as the root user.

Step 4: Create a VPC for the streaming instances

Create a VPC with two private subnets and two public subnets. The private subnets must be in two different availability zones. The public subnets must be in the same availability zones as the private subnets.
Create an Internet Gateway and attach it to the VPC.
Create a route table for public subnets that routes traffic to the Internet (0.0.0.0/0) via the Internet Gateway.
Create one NAT Gateway in each public subnet. Take note of the public IP address of the Elastic IP associated to the NAT Gateway, as you will need them later.
Create one route table for each private subnet that route traffic to the Internet via the NAT Gateway in the same availability zone.

Step 5: Create the Elastic fleet and stack

Create an IAM role for AppStream 2.0 that allows putting objects to the S3 bucket where recorded video files must be stored (prefix records/).
Follow the instructions at Create a fleet in the AppStream 2.0 documentation to create an Elastic fleet.
In Step 2: For Instance type, choose stream.standard.small. For Stream view, choose Desktop. For IAM role, choose the IAM role created in the step before. For Session script object in S3, choose the object session-scripts.zip that you uploaded to S3.
In Step 3: Choose the applications to associate with the Elastic fleet.
In Step 4: Disable Default Internet Access, and choose the appropriate VPC and private subnets, and its default security group.
Follow the instructions at Create a stack in the AppStream 2.0 documentation to create a stack.
In Step 1: For Fleet, choose the fleet created above.
In Step 2: You can enable Home Folders, if you wish.
In Step 3: You must disable copy/paste and file transfer from the streaming instance to the local workstation. You should also disable Print to local device, and Application settings persistence (not currently supported with Elastic Fleets).

Step 6: Prepare an EC2 instance and a user to use as system operator (preferably in another AWS account)

Create an IAM role for EC2 and attach the AWS-managed policy AmazonSSMManagedInstanceCore.
Create an EC2 instance from the latest Amazon Linux 2 AMI (or another AMI with the SSM agent preinstalled) with that IAM role attached. The new instance should appear in the list of managed nodes (see Start a session in the Session Manager documentation).
Create a user to use as a system operator. This can be an IAM user, an AWS SSO user, or another type of identity. If appropriate, create the AWS credentials needed to connect to the AWS management console and to make programmatic requests using the AWS CLI, such as a console password or an access key/secret key.
Attach an inline policy to the IAM user (or any other type of identity) to allow access only from the public IP addresses of the NAT Gateways.

Step 7: Test the solution as a system operator

In the AWS account where AppStream 2.0 resources have been created, create a streaming URL by using the AWS management console (see documentation). This simulates when the application that records the reason for access would do.
Connect to the streaming session using the streaming URL. You should see a new metadata.txt file in the S3 bucket where records are stored. Moreover, every 5 minutes, when the screen resolution changes, and at the end of the streaming session, you will see new video-{N}.mp4 files.
Launch the terminal in the streaming instance and configure the AWS credentials of the user you created in Step 6.
Establish a SSH connection using Session Manager with the EC2 instance you created in Step 6 by executing the following command: aws ssm create-session --target instance_id.
Open Google Chrome in the streaming instance and connect to the AWS management console with the user you created in Step 6.
Connect to the AWS management console from a browser on your local workstation. You should see that all requests fail with a “Permissions denied”.

Conclusion

In this article, I shared a solution, based on AppStream 2.0 and Session Manager, to record human activities on EC2 instances, managed middlewares and AWS, to reduce the risk of data exfiltration, and to capture the reason for access. If you built something to meet similar requirements, feel free to share by adding a comment.