Install & Configure Azure Monitor Agent (AMA) for Linux — Data Collection Rules & CEF Collector
What is the Azure Monitor Agent? 🤷🏽♂️
The Azure Monitor Agent (AMA) functions as a comprehensive data collection tool for Azure monitoring. It gathers a wide range of data, including performance metrics, logs, and events, from various platforms such as Windows, Linux, and different cloud environments. Once collected, this data is transmitted to Azure Monitor for detailed analysis. The AMA integrates seamlessly with other monitoring tools, centralizing data collection and analysis to provide a unified view of system performance and application health. This enables efficient monitoring and proactive issue resolution.
What is a Data Collection Rule? 🤷🏽♀️
Data Collection Rules (DCRs) in Azure provide a structured approach to data gathering and management. They define the criteria for collecting specific data, the processing methods, and the destinations for data analysis. Stored centrally within Azure, DCRs simplify the management of data collection configurations. In many cases, you can easily adjust settings via the Azure portal to create and deploy DCRs automatically. For more customized requirements, you can manually configure your own DCRs. Essentially, DCRs streamline the data collection process, ensuring that relevant data is efficiently directed to the appropriate analysis tools.
What is a CEF collector? 🤷🏽♂️
When dealing with log data from network and security devices, the Common Event Format (CEF) provides a structured and standardized method for data presentation, enhancing clarity and ease of interpretation. CEF simplifies the log data, making it more accessible for analysis tools like Microsoft Sentinel. While CEF is user-friendly for analysis, Sentinel also has the capability to parse and interpret less structured Syslog events during analysis. This ensures that all log data, regardless of its original format, is effectively processed and ready for detailed examination in your analysis tools.
Configuring Data Collection Rules (DCR) 🚀
Data Collection Rules (DCR) allow you specify what data should be collected, how to transform the data and where to send the data.
In order to configure a DCR we must have a Microsoft Sentinel environment. If you don’t have one already stop and go create one. Once in Sentinel find the Data connectors tab on the left-hand side. Once in Data connectors search for CEF and select Common Event Format (CEF) via AMA.
You’ll notice a few things from the image below:
- When a Data connector is enabled, it will light up in green.
- Under Data type you’ll see the table name the data connector is associated to. This will allow us to later on query the data using KQL.
- You’ll also see the out-of-the-box content that comes with the data connector. In this example we’re only seeing one query.
- The overview will also show you the level of traffic that the data connector ingesting. Since this a lab environment we’re not seeing much but, in a production environment we can see thousands.
Let’s now open our data connector to begin configuring our DCR. Click on “Create data collection rule”. Let’s provide a name, subscription and resource group.
Since this DCR is being created for Microsoft Sentinel it’s recommended to select the same Resource Group as Microsoft Sentinel
In terms of scope, we have the ability to select multiple or single objects. For this example, we’ll select our syslog machine.
If your machine is not a native Azure machine you MUST onboard the machine to Azure Arc. Please reference this documentation to onboard a machine to Azure Arc:
Onboard Linux Machine to Azure Arc
Now that we have our syslog machine select, we must select the correct facility and log level. In this example I’ll be sending my logs over LOG_LOCAL4 with a log level of LOG_INFO.
❓ Typically for facility levels you want to make sure you’re gathering all the security relevant data. If you’re using this to ingest firewall logs over syslog, please refer to the firewall vendor documentation.
Once you’ve validated your Data Collection Rule is configured correctly go ahead and select create.
Configuring Azure Monitoring Agent 👨🏽💻
Once the Data Collection Rule has been applied to the syslog machine, AMA will automatically be installed. This should take a few minutes and we can validate the install by checking Azure Arc.
Once in Azure Arc locate your syslog machine, once selected you’ll see on the left-hand side the Extensions tab. You should see the AMA agent downloaded; you can verify by checking the Status column. In my example it’s still downloading, let’s give it a few minutes to bake and check back.
After 5 minutes the status changed to Succeeded.
Configure CEF Collector 🔨
Now that we have our Data Collection Rule and Azure Monitor Agent installed on our syslog machine our last step is to configure it as CEF Collector. This can be done by running the following command to download the configuration script.
sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.pyPlease edit the command to run the correct version of python that you have installed on your syslog server.
Validate DCR, AMA and CEF Configuration 🤞🏽
Now that we’ve completed all the initial steps, we need to ensure that none of our configuration files are corrupted.
Validate Syslog Configuration File (AMA and CEF) 📌
Syslog must be able to receive events from another source such as Firewall. We can verify by checking if port 514 is open on both TCP and UDP. Typically, once AMA is installed it will automatically be configured. Run this command to check the syslog conf file.
Let’s also check that all conf files are included. Typically, it’ll be located at the bottom of the file under /etc/rsyslog.d/*conf.
If you made any changes to the rsyslog file it’s always best to restart the syslog service. You can do so by running this command: sudo systemctl restart syslog
If the Azure Monitor Agent was installed correctly, we should a file named 10-azuremonitoragent-omfwd.conf.
The Azure Monitor Agent installs an output configuration file which specifies the way events file between the syslog daemon and AMA. For syslog (most Linux Distrubutions) the file is /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf.
Validate Data Collection Rule (DCR) 🧭
We need to ensure that the DCR we created in Microsoft Sentinel has been applied to our syslog machine. This information is stored in a json file under /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks.
We can inspect the file by using the cat command.
sudo cat /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/jsonfilename.json
The JSON file tells AMA what type of events should be ingested and forwarded to Microsoft Sentinel.
Now.. Does it actually work? 🤔
We’ve done everything right… hopefully! Let’s send some test logs!! We can use the logger command to send out first log to Microsoft Sentinel. Copy this command to your syslog machine.
logger -p local4.info -P 514 -n 127.0.0.1 --rfc3164 -t CEF "0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time"
Please note, my DCR is configured to accept traffic only coming from Log facility level 4 info. Based on your DCR configuration you may need to change this in the logger command.
To validate syslog received our message we can use a few different options but the simplest would-be using grep since we know what we’re looking for in the log. In a production environment we might want to tail the syslog so we can see data flow in real time.
grep -w Mock-test /var/log/syslog
We confirmed that syslog picked up the message let’s jump over Microsoft Sentinel and query the table CommonSecurityLog.
Closing Thoughts 💭
Microsoft Sentinel truly simplifies the process of handling logs. Getting those “3rd party” logs into Sentinel? Surprisingly easy! It’s impressive how effortlessly it manages what could be a complex task.
Microsoft Sentinel really surprises with how smoothly it handles tasks like log ingestion. But remember, this is just the beginning of what Sentinel can do. There’s a whole lot more waiting to be explored!
Sentinel’s strength lies in making things easier without adding unnecessary complications. It’s a tool that simplifies the complex, offering a range of features that streamline data management and analysis. There’s a lot more to explore and appreciate within Microsoft Sentinel!
