Easy Security Log Analysis and Visualization with Amazon Security Lake

Security Lake has been officially available since last May. In case you’re not familiar with it, think of Security Lake as a data lake that’s highly secure, where you can store all sorts of security logs.

With Security Lake, you get the benefit of automatically streaming and storing AWS Native security logs and data from various third-party solutions directly into your chosen S3 target. Here’s the another essential feature of Security Lake — it collects diverse security logs in an unified format that complies with the OCSF schema*. With the feature, you can reduce the time and effort to unify the schema.

(*For those not in the know, OCSF provides a standard schema for common security events.)

For fine-grained data management, LakeFormation comes into play, taking the reins as the admin for handling data permissions. You can also leverage AWS ETL solutions, including Glue and Athena, along with BI solutions like QuickSight, to gain a crystal-clear understanding of your resources’ security status.

Let’s take a sneak peek at the steps involved in analyzing Security Lake data:

Here is the architecture for Security Lake. You can integrate multiple sources, such as AWS native logs and various third-party logs, with Security Lake. Once the logs are collected in S3 through Security Lake, we can use Glue, Athena, and QuickSight to parse and visualize the data.

After enabling Security Lake and configuring the sources properly, the data is stored in specified S3 paths. For instance, I enabled AWS Native Logs, as shown in the photo above, and now I can access all the data from the path.

As mentioned earlier, 3rd-party data is also integrated into Security Lake in specified S3 paths. To ensure better data management, I separated the paths accordingly.

In order to view and analyze the data in the Route53 folder, I performed a simple data parsing process with Athena after creating initial table with Glue crawler. Please note that you need to grant Glue role the permission to access S3 data with LakeFormation.

During the parsing,I casted the ‘time’ column as a timestamp format to leverage it in QuickSight with the appropriate format. Additionally, for columns like ‘answers_type’ which are in an array format, I used the element_at function to access the elements. Here’s the query that I used to parse Route53 data.

SELECT 
 metadata.product.name AS product_name,
 metadata.product.feature.name AS product_feature_name,
 src_endpoint.vpc_uid AS src_endpoint_vpc_uid,
 src_endpoint.ip AS src_endpoint_ip,
 src_endpoint.port AS src_endpoint_port,
 src_endpoint.instance_uid AS instance_uid,
 CAST(from_unixtime(time / 1000) AS timestamp) as time,
 query.hostname AS query_hostname,
 query.type AS query_type,
 query.class AS query_class,
 element_at(answers, 1).type AS answers_type,
 element_at(answers, 1).rdata AS answers_rdata,
 element_at(answers, 1).class AS answers_class,
 connection_info.protocol_name AS connection_info_protocol_name,
 connection_info.direction AS connection_info_direction,
 dst_endpoint.instance_uid AS dst_endpoint_instance_uid,
 dst_endpoint.interface_uid AS dst_endpoint_interface_uid,
 class_name
 class_uid,
 category_name,
 activity_name, 
 region,
 accountid,
 cast(date_parse(eventday,'%Y%m%d') as date) AS eventday
FROM "amazon_security_lake_bucket"."route53" limit 10;

Now that the data is ready, it’s time to set up QuickSight. Since the data is controlled by LakeFormation, additional permissions need to be granted to add datasets on QuickSight.

To grant QuickSight user the necessary permissions for all the tables I created, I simply used the LakeFormation CLI command below.

aws lakeformation grant-permissions \ 
 - principal DataLakePrincipalIdentifier="arn:aws:quicksight:region:account:user/user_name"
 - permissions "ALL" \
 - resource '{ "Table": { "DatabaseName": "amazon_security_lake_glue_db_ap_northeast_2", "TableWildcard": {} } }'\
 - profile sl

Once the steps above are completed, you can add dataset from QuickSight and start creating your security dashboard.

You can now create a 360-degree view of the security dashboard by leveraging various features such as filters, navigation actions, etc.

If you have any questions regarding configuring Amazon Security Lake or the Security Dashboard, feel free to leave a comment!