Sending system logs from EC2 to Amazon Security Lake with Cribl Edge
AWS has recently introduced Amazon Security Lake for the purpose of collecting and analyzing data from AWS services. However, when you and your team require a way to gather system logs, which include application and OS logs, and store them in the Security Lake for further utilization, Cribl Edge offers a complete solution for sending these logs to the Security Lake more efficiently and quickly.
Cribl Edge
Cribl Edge, which can be installed on servers either as a stand-alone application or as an agent, sends logs in real-time to the desired destination whenever logs are generated in files.
Cribl has been introduced as a partner on the official AWS blog, and the key aspects you can achieve through Edge are as follows:
- Equipped with connectors to send data not only to Security Data Lake but also to other destinations like S3. This enables the simultaneous sending and storage of raw events for archival purposes.
- Provides a GUI environment in Cribl Edge to easily and intuitively build data pipelines
- Offers complete monitoring of data flow
- Debugging can be done within the Cribl Edge solution to verify and validate the transformed data before it is delivered to the destination when performing data transformation tasks.
Test Process
The process of building the date pipeline using Cribl Edge is as follows:
- Install Cribl Edge on the DVWA server.
- Configure Source connectors for each log file.
- Set permissions before transmitting to Amazon Security Lake in Cross Account and configure Destination connectors.
- Build each pipeline to transfer data from Source to Destination.
Test Environment
- Instance Type : t3.medium
- OS : Amazon Linux v2
- Data Streaming Solution : Cribl Edge v4.1.3
- Data : Web Access log, Messages, History, Web Secure log, Error log
1. Install Cribl Edge
Installing Cribl Edge, as well as tasks like configuring Edge settings and registering PATH variables, is necessary. The script provided below is used to simplify this installation process.
### Installing git for distribute mode to deploy configurations
#sudo yum -y install git jq
### Installing Cribl && move the directory under /opt/
if [ ! -d /opt/cribl ]; then
sudo curl -Lso - $(curl https://cdn.cribl.io/dl/latest-x64) | tar zxv && mv cribl /opt/
else
echo "Cribl is already installed"
fi
### Making variable for Cribl directory
if [ -d /opt/cribl ]; then
# Change to Cribl Edge mode (Single Instance)
cd /opt/cribl/bin
# Update Cribl Edge Setting
./cribl mode-edge -H 0.0.0.0 -p 9420
# Register Path variable
echo "export CRIBL_HOME=/opt/cribl" >> /etc/profile
source /etc/profile
# echo "path to Cribl : $CRIBL_HOME"
# Starting Cribl and boot-start
sudo $CRIBL_HOME/bin/cribl start
# sudo $CRIBL_HOME/bin/cribl boot-start enable -m initd
else
echo "Cribl is not installed, failed to run Cribl"
fiThe default port for Cribl Edge is 9420, and you can access to it immediately after the installation is complete.
2. Configure Source Connectors
After logging in to Cribl Edge, navigate to “More” > “Source” in the top menu to select “File Monitor” to choose data in log files from the server.
By clicking “Add Source” on the upper right corner, you can create a new connector and configure the necessary settings for the connector. In the options, enter the path and file name where the data is generated in ‘Search path’ and ‘File name.’ Then, you can see the connector you created listed in the form of the list
Navigate to Cribl’s ‘‘Live Data’’ menu to observe data being generated in real-time. The command entered for testing is stored in the history file and can be verified through the Live Data menu.
3. Configure Destination Connectors
To transmit the data to the destination, navigate to “More’’ > ‘‘Destination’’ in the top menu and select the Amazon Security Lake connector.
By clicking ‘‘Add Destination’’ on the upper right corner, you can create a new connector and configure the necessary settings for the connector. As key configuration options, provide the ‘S3 bucket name’, ‘region’, ‘AccountID’, ‘Customer source name’ and ‘Assume role.’ Afterward, you can observe the connector you created being organized in the form of a list.”
Before being transmitted to Amazon Security Lake, the data needs to be converted to Parquet format. To achieve this, the schema should be stored within Cribl
Through the ‘‘status’’ menu, you can test the communication with the destination in advance. In case of communication failure, you can review the failure sign and message along with the failure reason.
4. Build Data Pipeline
In this test, we used the Quick Connect feature to easily build a data pipeline for transmitting data from the server to Amazon Security Lake. This feature allows you to connect Source and Destination using a drag-and-drop approach, requiring just a few mouse clicks to build data pipelines.
Before the data is transmitted to Amazon Security Lake, it’s necessary to transform the data to align it with the required fields. Pre-conversion debugging can be performed within Cribl.
The key functions below will be essential before being forwarded to Amazon Security Lake, and within Cribl, these tasks can be performed code-lessly.
After building the data pipelines, verify that the data is successfully stored.
Conclusion
In this blog post, we explained the process of sending system logs to Amazon Security Lake using Cribl Edge. If you’re interested in learning the exact procedure, please reach out to cribl@megazone.com or feel free to leave comments. Thank you for reading!
- To try out Cribl’s features, visit https://sandbox.cribl.io/
- For immediate access and use of Cribl, go to https://cribl.cloud/
- To install Cribl in your own environment, visit https://cribl.io/download/
#MegazoneCloud #aws #cloud #devops #AmazonSecurityLake #Cribl #CriblEdge #Cribl.Cloud
