Malicious Activity Detection in Oracle Cloud using Falco and OCI OpenSearch — Part 2

In part 1, we learned about Falco, a cloud native security tool designed to help you with threat detection, especially on Linux and cloud native systems. We also learned how to install it on a compute instance running Oracle Linux in Oracle Cloud (OCI)and then ship detected events to OCI Logging service.

Now, some users may already be familiar with other observability and logging tool e.g. Splunk, DataDog etc. In this article, we are going to continue the previous exercise but this time, we are going to ship Falco events from a compute instance in OCI to another observability tool and we are going to use OpenSearch since OCI already has a managed OpenSearch service:

Shipping Falco events to OpenSearch via syslog and fluentd

Now, we could have used Falco’s standard output and Fluentd to then ship the events to OpenSearch. But Falco also has this sidekick sub-project which can ship events directly to different outputs and has a UI as well, which we’ll explore in a future post. So, we are going to try that instead.

Shipping events to OCI OpenSearch

Create a compute instance and install Falco as described in the previous article and create an OpenSearch cluster. When you create the OpenSearch cluster, you’ll be prompted for a username and password. You’ll use these values later.

Adjust your network security rules (NSG, security list) and from your compute instance, ensure you can reach the OpenSearch API endpoint:

nc -v 10.0.122.192 -p 9200

Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to 10.0.122.192:9200.

We can now install falcosidekick:

sudo mkdir -p /etc/falcosidekick
wget https://github.com/falcosecurity/falcosidekick/releases/download/2.28.0/falcosidekick_2.28.0_linux_amd64.tar.gz && sudo tar -C /usr/local/bin/ -xzf falcosidekick_2.28.0_linux_amd64.tar.gz

Create a config file for the sidekick so it can start pushing events to OCI OpenSearch and save it to /etc/falcosidekick/config.yaml:

elasticsearch:
  hostport: "" # https://<private_ip_of_opensearch_api>:9200 
  index: "falco" # index (default: falco)
  suffix: "daily" # date suffix for index rotation : daily (default), monthly, annually, none
  mutualtls: false # if true, checkcert flag will be ignored (server cert will always be checked)
  checkcert: false # check if ssl certificate of the output is valid (default: true)
  username: "" # username to authenticate to OCI OpenSearch service
  password: "" # password to authenticate to OCI OpenSearch service

Follow the rest of the Falco guide and create a Systemd service:

vi /usr/lib/systemd/system/falcosidekick.service

And enter the following:

[Unit]
Description=Falcosidekick
Documentation=https://github.com/falcosecurity/falcosidekick.git
After=network.target

[Service]
Type=simple
Restart=always
User=root
ExecStart=/usr/local/bin/falcosidekick -c /etc/falcosidekick/config.yaml

[Install]
WantedBy=multi-user.target
Alias=falcosidekick.service

Then enable the services:

systemctl enable falcosidekick
systemctl start falcosidekick

And check its status:

systemctl status falcosidekick
● falcosidekick.service - Falcosidekick
   Loaded: loaded (/usr/lib/systemd/system/falcosidekick.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-10-13 16:38:48 AEDT; 17min ago
     Docs: https://github.com/falcosecurity/falcosidekick.git
 Main PID: 111686 (falcosidekick)
    Tasks: 5 (limit: 22531)
   Memory: 22.0M
   CGroup: /system.slice/falcosidekick.service
           └─111686 /usr/local/bin/falcosidekick -c /etc/falcosidekick/config.yaml

We must now check if the sidekick can talk to OCI OpenSearch. Run the following command:

curl -XPOST "http://localhost:2801/" -d'{"output":"16:31:56.746609046: Error File below a known binary directory opened for writing (user=root command=touch /bin/hack file=/bin/hack)","hostname": "localhost", "priority":"Error","rule":"Write below binary dir","time":"2019-05-17T15:31:56.746609046Z", "output_fields": {"evt.time":1507591916746609046,"fd.name":"/bin/hack","proc.cmdline":"touch /bin/hack","user.name":"root"}}'

And then check the service status:

Oct 13 16:38:48 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:38:48 [INFO]  : Falco Sidekick version: 2.28.0
Oct 13 16:38:48 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:38:48 [INFO]  : Enabled Outputs : [Elasticsearch]
Oct 13 16:38:48 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:38:48 [INFO] : Falco Sidekick is up and listening on :2801
Oct 13 16:39:15 o-jtzjvw falcosidekick[111686]: 2023/10/13 16:39:15 [INFO]  : Elasticsearch - Post OK (201)

If you see the HTTP code 201, then Falco sidekick is working.

We now need to update Falco itself to send its events to the sidekick. Edit the falco config:

vi /etc/falco/config.yaml

And change the following:

json_output: true

http_output:
  enabled: true
  url: "http://localhost:2801/"

Restart the sidekick service too:

sudo systemctl restart falcosidekick

Testing the system

We are now ready to test the entire system. Let’s take an action that will generate a Falco event as we did in the previous article:

sudo cat /etc/shadow > /dev/null

Check if Falco detected it:

Oct 13 17:14:26 o-jtzjvw falco[101975]: {"hostname":"o-jtzjvw","output":"17:14:26.479321063: Warning Sensitive file opened for reading by non-trusted program (file=/etc/shadow gparent=bash ggparent=sshd gggparent=sshd evt_type=openat user=root user_uid=0 user_loginuid>

Check the Falcosidekick status. There should be one that matches the time above:

Oct 13 17:14:26 o-jtzjvw falcosidekick[111686]: 2023/10/13 17:14:26 [INFO]  : Elasticsearch - Post OK (201)

And finally, let’s access the OCI OpenSearch Dashboard:

 ssh opc@<bastion_ip> -L 5601:<opensearch_dashboard_private_ip>:5601

Access the OpenSearch Dashboard in your browser: https://localhost:5601/ and login with the username and password. Navigate to Stack Management > Index Patterns and create an Index Pattern for Falco:

Once the Index Pattern is created, navigate to OpenSearch Dashboards > Discover, then select the index pattern you just created for Falco:

You should now be able to see some events:

Falco Events in OpenSearch Dashboard

The earlier events were the tests we were doing to make sure things are working but the latest one is the one that triggered a Falco event:

And voila!

Summary

In the previous and this article, we’ve shown that Falco can be used to detect potential security threats that may occur on compute instances. Even though these are normal compute instances and not part of Kubernetes, you can still use Falco to detect threats.

Once detected, we then ship those events to OCI OpenSearch service via the Falcosidekick. With the OpenSearch Dashboard, we can then construct the necessary visualizations and dashboards to monitor and understand security threats happening in our infrastructure.

With this in mind, it’s not hard to imagine a fleet of compute instances or instance pools shipping their Falco events to OCI OpenSearch as each instance will have Falco and its sidekick running:

In a future post, we’ll look at achieving similar outcomes but in a Kubernetes environment such as OKE.