Advancing AWS EC2 Incident Response: Addressing Dormant Tracked Connections

In my previous blog post on AWS Incident Response, which I recommend reading first; I explored the automation of containment strategies for AWS entities, including IAM users, roles, AWS Lambda functions, and EC2 instances. The journey uncovered a complex approach necessary for EC2 containment, driven by AWS security group connection tracking nuances. Building upon that foundation, this post delves deeper into addressing the challenges of dormant connections.

The Challenge of Dormant Connections

AWS security group connection tracking plays a crucial role in ensuring EC2 instances communicate without interruption, even when security rules are modified. However, this beneficial feature complicates incident response efforts, especially when dealing with dormant connections that don’t exhibit continuous traffic, such as those seen in “low and slow” attack strategies or Command and Control (C2) channels via SSH. The persistence of these tracked connections, despite the removal of all relevant security group rules, underscores the necessity for a sophisticated approach to incident containment, going beyond the simple adjustment of security group configurations.

Developing an automated solution for EC2 instance containment, I
hit a snag with dormant, tracked connections. My initial strategy — converting tracked to untracked connections before dropping security group rules — was effective for active connections. Yet, it faltered with inactive ones, particularly problematic for SSH or reverse shell scenarios. The workaround involved a waiting period, hoping for activity that would trigger the conversion process. This method, however, was imperfect.

AWS’s Feature Update

Just before the new year, AWS introduced a game-changer: a
configurable idle timeout for connection tracking, an Elastic Network Interfaces (ENI) configuration option. While not designed with security as its primary intent, this feature offered an elegant solution to my problem. By setting an idle timeout, I can ensure the automatic termination of inactive connections, thus enhancing the containment strategy’s effectiveness. This feature enables customization of the duration that connection tracking entries remain idle before being removed, offering better control over network resources and security by addressing challenges associated with dormant connections. This adjustment aims to improve efficiency and security for AWS customers, ensuring more robust and adaptable network traffic management. Here is a link to the AWS feature introduction blog.

In the AWS EC2 incident response context, it’s crucial to understand the interconnected roles of security groups and Elastic Network Interfaces (ENIs) in managing instance traffic flow. Security groups act as virtual firewalls for EC2 instances, controlling both inbound and outbound traffic at the ENI level. Each ENI is associated with a security group, and together, they determine how traffic is allowed or denied, significantly impacting incident response strategies. The introduction of configurable idle timeouts for connection tracking at the ENI level enhances our ability to manage these connections more effectively, particularly for dormant connections. This feature allows for a more nuanced approach to containment, ensuring that even connections without continuous activity can be addressed.

Real-world Application Examples

Let’s dissect a few examples to illustrate the solution’s breadth, keeping in mind that setting the idle timeout for connection tracking is a nuanced operation performed at the ENI level. This setting indirectly influences security group behavior, ensuring that even dormant connections without continuous activity can be effectively managed and contained:

1. Simple Case: An EC2 instance with an open security group rule (quad zero) isn’t tracked. Swapping the security group for a containment group with no rules isolates the instance.
2. Moderate Complexity: An instance engaged in crypto mining, with specific inbound/outbound rules, represents a tracked connection. Continuous communication ensures that swapping security groups effectively contains the instance, thanks to the conversion of tracked to untracked connections.
3. High Complexity: A reverse shell scenario with sporadic activity poses the most formidable challenge. Here, setting the idle timeout for TCP connections to 60 seconds (the minimum setting) for each ENI, followed by the security group swap, ensures the containment of the compromised instance. This operation at the ENI level is critical for addressing continuous and sporadic traffic, offering a comprehensive containment strategy.

Below is a snippet demonstrating how to configure the TcpEstablishedTimeout for an Elastic Network Interface (ENI), a crucial setting for optimizing AWS EC2 incident response strategies

enis = client.describe_network_interfaces(Filters=[{'Name': 'attachment.instance-id', 'Values': [entity['entity_value']]}])

for eni in enis['NetworkInterfaces']:
    eni_id = eni['NetworkInterfaceId']
    client.modify_network_interface_attribute(
        NetworkInterfaceId=eni_id,
        ConnectionTrackingSpecification={
            'TcpEstablishedTimeout': 60
        }
    )

Conclusion

While the new AWS feature hasn’t simplified the solution, it significantly extends its coverage, including previously unaddressable edge cases. This evolution underscores the continuous need for adaptation in the cloud security landscape.

I invite you to join the conversation and contribute to the open-source project on GitHub: AWS Incident Response. I’ve developed a turnkey, automated solution designed for seamless implementation, aiming to simplify the complexity of AWS incident response.