Sudo Login Alerting in Google Cloud Platform

Enterprise operation teams are in need of alerting mechanism for tracking the sudo user logins to critical compute instances. Even though all the IAM policies and security measures in place, but some of the legitimate users having access to machine can execute the the sudo login as root or any other functional user to perform certain operations. In these cases even granular IAM policy setting will not help to prevent or filter the same.

There are few ways to control the sudo login to machine. But most of the use cases we need to allow automation scripts, tools and other operations users to login as sudo functional user to perform certain actions.

In that case preventing the overall sudo login machine facility will not help. If conditionally enabling of users and other placeholders needed then options is to modify the os specific files locally based on the user id’s. But these changes will be overwritten once the OS patching is done on that particular instance or machine. So this is also not an efficient option to proceed further.

Considering all facts stated above, operation teams need a mechanism, which is simple, efficient, easy to implement and easy to maintain. Also should not be overwritten by any operating system patching activity as well. It should handle all the cases where sudo will be used. Below are few examples of sudo command usage:

sudo -i //login as root user
sudo su //login as switch user
sudo su  -  // login as root user
sudo su - tbaadm // login as special user by name tbaadm

All the above scenarios of logging in as sudo user can be captured and alerting can be triggered on the event in near realtime.

Google Cloud Platform (GCP) offers excellent tools for streaming and alerting based on logs generated. We are going to use below components to build a sudo login alerting mechanism in GCP.

1. Google ops agent

2. Sudoers custom configuration.

3. GCP log based alerting policy.

You can follow below step by step instruction to build a sudo login alerting in GCP of linux based google compute engine instance.

Step 1: Install the GCP ops agent in your Google compute engine(GCE) instance. Execute below commands in your GCE instance.

1. curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
2. sudo bash add-google-cloud-ops-agent-repo.sh - also-install
3. sudo systemctl status google-cloud-ops-agent

Step 2: You can use the ops agent configuration file and modify it as below to define the custom log streaming via ops-agent. To perform this file operation you should have the root privileges.

Filename: /etc/google-cloud-ops-agent/config.yaml

logging:
  receivers:
    sudo-cmd-log:
      type: files
      include_paths:
      - /var/log/sudo.log
  service:
    pipelines:
      sudo-cmd-pipeline:
        receivers: [sudo-cmd-log]

Once modification is complete, you can restart the ops-agent to start the streaming process. You can use below commands to restart and to check the status of ops-agent process.

sudo systemctl restart google-cloud-ops-agent 
sudo systemctl status google-cloud-ops-agent

Now ops agent will look for the /var/log/sudo.log file from local GCE instance and stream the log file into GCP logging dashboard.

Step 3: You can use the /etc/sudoers.d directory to create a new file for creating the sudo.log with authentication information. So that this log will be populated with necessary information on the user who executed the sudo command along with timestamp etc.,

Filename: /etc/sudoers.d/new_sudoers

you can use visudo -f /etc/sudoers.d/new_sudoers OR sudo visudo -f /etc/sudoers.d/new_sudoers to edit the file. Place below content on to the /etc/sudoers.d/new_sudoers file. To perform this file operation you should have the root privileges.

Defaults        syslog=auth
Defaults        log_year, logfile=/var/log/sudo.log

By default /etc/sudoers file will read the content of the directory /etc/sudoers.d , by doing this way we can keep our customization separate from the original file. And it will not be overwritten by an OS upgrade or patching changes.

After these changes, system will capture every sudo command execution on to /var/log/sudo.log file.

P.S: you can put the /var/log/sudo.log on rotation so that each day file can be backed-up as tar.gz file and a new file can be created for each day. This can be scheduled as per the frequency and need of the ops team as required. Old backup zip files can be deleted as per space utilization needs.

Step 4: Execute below steps to create an alerting policy to trigger an alert when there is an entry created on sudo.log and streamed as a sudo-cmd-log file onto GCP logging (stackdriver) space.

Navigate GCP console →Logging →Logs Explorer. Select log name as “ sudo-cmd-log” and apply.

Then click on “create alert” to create log based alerting policy.

It will open up a window popup in same page to fill in all the details required on this new alerting policy as below and click next.

Below second step indicates which log file used for this alerting. Here you can specify any custom requirements such only root related login to be selected for alerting or any specific special user login related alerting can be done.

For this use case we are only selecting the entries which are related to root user login.

Below third step indicates notification time frame and incident autoclosure duration for our alerting. User can specify the notification frequency based on the instance criticality as it differs from use case to use case.

Here we will go with a minimum of 5 minutes time between notification. This means GCP will look for this log every 5 minutes to check entries and trigger the alert accordingly.

Once the alert is triggered an incident will be opened automatically. If no action is taken on incident then it will be auto closed based on duration set in below screen. Click on next to navigate to final screen.

Below final screen of alerting policy page, you will need to choose appropriate notification channel to receive this alert message.

There are many notification channels supported by GCP such as Email, SMS, Pub/Sub, PagerDuty , Slack , Mobile Devices and Webhooks. Here will go with creating Email notification channel to configure this alerting policy.

Click on the Notification channels. It will open up a drop down where in you can use the existing notification channel or create a new notification channel to receive the notifications.

Click on “Manage notification Channels”. It will open up the notification channel page in new window to create new notification channel.

Under Email section → Click on “ADD NEW” option.

Enter the email address and display name for the notification channel and click on save.

You should navigate back to the old window page where alerting policy details were entered and click on refresh button to see the new notification channel configured as below and select your notification channel and click OK.

Final page of alerting policy would look like below. Review all the details and click on save.

Step 5: Perform root user login from your GCE instance using sudo command such as sudo -i or sudo su — . It will trigger the alert and you should see email like below on your email id configured as part of the notification channel.

We have now successfully configured a sudo login alerting mechanism on Google compute instance. Users can further enhance the log filtering conditions to include specific special user login using sudo to configure the alerts as required.

Now operations team can monitor and configure the alerting on the sudo logins executed on Google compute engine (GCE) instances.

Thanks to Srinivas Totapally & Ramakrishnan Sankarasubramanian for guidance and support on this article.