Malicious Activity Detection in Oracle Cloud using Falco and OCI OpenSearch — Part 2
In part 1, we learned about Falco, a cloud native security tool designed to help you with threat detection, especially on Linux and cloud native systems. We also learned how to install it on a compute instance running Oracle Linux in Oracle Cloud (OCI)and then ship detected events to OCI Logging service.
Now, some users may already be familiar with other observability and logging tool e.g. Splunk, DataDog etc. In this article, we are going to continue the previous exercise but this time, we are going to ship Falco events from a compute instance in OCI to another observability tool and we are going to use OpenSearch since OCI already has a managed OpenSearch service:
Now, we could have used Falco’s standard output and Fluentd to then ship the events to OpenSearch. But Falco also has this sidekick sub-project which can ship events directly to different outputs and has a UI as well, which we’ll explore in a future post. So, we are going to try that instead.
Create a compute instance and install Falco as described in the previous article and create an OpenSearch cluster. When you create the OpenSearch cluster, you’ll be prompted for a username and password. You’ll use these values later.
Adjust your network security rules (NSG, security list) and from your compute instance, ensure you can reach the OpenSearch API endpoint:
nc -v 10.0.122.192 -p 9200
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.122.192:9200.
We can now install falcosidekick:
sudo mkdir -p /etc/falcosidekick
wget https://github.com/falcosecurity/falcosidekick/releases/download/2.28.0/falcosidekick_2.28.0_linux_amd64.tar.gz && sudo tar -C /usr/local/bin/ -xzf falcosidekick_2.28.0_linux_amd64.tar.gz
Create a config file for the sidekick so it can start pushing events to OCI OpenSearch and save it to /etc/falcosidekick/config.yaml:
elasticsearch:
hostport: "" # https://<private_ip_of_opensearch_api>:9200
index: "falco" # index (default: falco)
suffix: "daily" # date suffix for index rotation : daily (default), monthly, annually, none
mutualtls: false # if true, checkcert flag will be ignored (server cert will always be checked)
checkcert: false # check if ssl certificate of the output is valid (default: true)
username: "" # username to authenticate to OCI OpenSearch service
password: "" # password to authenticate to OCI OpenSearch service
Follow the rest of the Falco guide and create a Systemd service:
vi /usr/lib/systemd/system/falcosidekick.service
And enter the following:
[Unit]
Description=Falcosidekick
Documentation=https://github.com/falcosecurity/falcosidekick.git
After=network.target
[Service]
Type=simple
Restart=always
User=root
ExecStart=/usr/local/bin/falcosidekick -c /etc/falcosidekick/config.yaml
[Install]
WantedBy=multi-user.target
Alias=falcosidekick.service
Then enable the services:
systemctl enable falcosidekick
systemctl start falcosidekick
And check its status:
systemctl status falcosidekick
● falcosidekick.service - Falcosidekick
Loaded: loaded (/usr/lib/systemd/system/falcosidekick.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-10-13 16:38:48 AEDT; 17min ago
Docs: https://github.com/falcosecurity/falcosidekick.git
Main PID: 111686 (falcosidekick)
Tasks: 5 (limit: 22531)
Memory: 22.0M
CGroup: /system.slice/falcosidekick.service
└─111686 /usr/local/bin/falcosidekick -c /etc/falcosidekick/config.yaml
We must now check if the sidekick can talk to OCI OpenSearch. Run the following command:
curl -XPOST "http://localhost:2801/" -d'{"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","hostname": "localhost", "priority":"Error","rule":"Write below binary dir","time":"2019-05-17T15:31:56.746609046Z", "output_fields": {"evt.time":1507591916746609046,"fd.name":"/bin/hack","proc.cmdline":"touch /bin/hack","user.name":"root"}}'
And then check the service status:
Oct 13 16:38:48 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:38:48 [INFO] : Falco Sidekick version: 2.28.0
Oct 13 16:38:48 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:38:48 [INFO] : Enabled Outputs : [Elasticsearch]
Oct 13 16:38:48 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:38:48 [INFO] : Falco Sidekick is up and listening on :2801
Oct 13 16:39:15 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:39:15 [INFO] : Elasticsearch - Post OK (201)
If you see the HTTP code 201, then Falco sidekick is working.
We now need to update Falco itself to send its events to the sidekick. Edit the falco config:
vi /etc/falco/config.yaml
And change the following:
json_output: true
http_output:
enabled: true
url: "http://localhost:2801/"
Restart the sidekick service too:
sudo systemctl restart falcosidekick
Testing the system
We are now ready to test the entire system. Let’s take an action that will generate a Falco event as we did in the previous article:
sudo cat /etc/shadow > /dev/null
Check if Falco detected it:
Oct 13 17:14:26 o-jtzjvw falco[101975]: {"hostname":"o-jtzjvw","output":"17:14:26.479321063: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=bash ggparent=sshd gggparent=sshd evt_type=openat user=root user_uid=0 user_loginuid>
Check the Falcosidekick status. There should be one that matches the time above:
Oct 13 17:14:26 o-jtzjvw falcosidekick[111686]: 2023/10/13 17:14:26 [INFO] : Elasticsearch - Post OK (201)
And finally, let’s access the OCI OpenSearch Dashboard:
ssh opc@<bastion_ip> -L 5601:<opensearch_dashboard_private_ip>:5601
Access the OpenSearch Dashboard in your browser: https://localhost:5601/ and login with the username and password. Navigate to Stack Management > Index Patterns and create an Index Pattern for Falco:
Once the Index Pattern is created, navigate to OpenSearch Dashboards > Discover, then select the index pattern you just created for Falco:
You should now be able to see some events:
The earlier events were the tests we were doing to make sure things are working but the latest one is the one that triggered a Falco event:
And voila!
Summary
In the previous and this article, we’ve shown that Falco can be used to detect potential security threats that may occur on compute instances. Even though these are normal compute instances and not part of Kubernetes, you can still use Falco to detect threats.
Once detected, we then ship those events to OCI OpenSearch service via the Falcosidekick. With the OpenSearch Dashboard, we can then construct the necessary visualizations and dashboards to monitor and understand security threats happening in our infrastructure.
With this in mind, it’s not hard to imagine a fleet of compute instances or instance pools shipping their Falco events to OCI OpenSearch as each instance will have Falco and its sidekick running:
In a future post, we’ll look at achieving similar outcomes but in a Kubernetes environment such as OKE.