Monitoring SSH login via Slack

Most companies secure their ssh logins through a variety of mechanism, be it 2-factor, key-based, host-based or a combination of these. Even after securing this, it will help to know when someone is logging in or logging out of the different nodes that you are managing. It is quite impractical to keep looking at the ssh secure logs on the server every now and then or even at the end of a day as attacks come in all the time, especially on cloud environments.

Even if you implement schemes like Fail2ban which allow you to configure banning IP addresses that try to login via brute force, it is a good practice (And warranted in PCI findings) to know who is logging into the nodes.

Slack is a very common communication tool that we use day-to-day and we wanted to know the user authentication patterns and also to know who is logging in/out of our servers. With a combination of these two, we decided to create a plugin that will allow SSH logins to be sent to a Slack channel that we all monitor so we know of any intrusions or valid logins.

To achieve this, we need to configure 2 different items:

• Slack: To create a webhook and let Slack know where it can receive its notifications
• SSH Node(s): Create a script on each of the nodes and let it know where to post the information about logins.

Slack Configuration:

You will need to create an incoming webhook on Slack. The complete instructions are here but we will go through the absolute basics to get you going.

• Create a slack app: Click here to create a new Slack App. Give it a name that you can recognize and configure the workspace where this needs to live. If you are already signed into Slack on the web, then most of this will be filled up for you
• Enable Incoming webhooks: After going into your Slack app, Activate the Activate Incoming Webhooks toggle to switch it on in the Settings page.
• Create a new incoming webhook for our setup: Once you are in the “Incoming Webhooks” section, click on the “Add configuration” button to create a new webhook. Slack changes this quite often, so this might not look the same if you are looking at this later , but please post back if its different and we can update this. At the time of writing, the page looks like the below:

• Once you select a channel from the list, you can click on the button to add the webhook. You will be presented with a Webhook URL, lets call this $WEBHOOK. Remember to write this down as we will use it on our SSH script.

SSH Configuration:

Once the Slack configuration is complete, go into the node where you need the logins monitored. Follow the below steps.

• Create a folder called /var/ops . This can be anywhere on your system though.
• Create a file called notify.sh with the following contents

#!/bin/bash

url="https://hooks.slack.com/services/XXXXX/YYYYY/ZZZZ"  #[Paste you slack webhook url that you got from the earlier setup]
channel="#bastion_security"  #[Slack channel name where you would like to receive the notifications]

host="`hostname`"

content="\"attachments\": [ { \"mrkdwn_in\": [\"text\", \"fallback\"], \"fallback\": \"SSH login: $USER connected to \`$host\`\", \"text\": \"SSH login to \`$host\`\", \"fields\": [ { \"title\": \"User\", \"value\": \"$USER\", \"short\": true }, { \"title\": \"IP Address\", \"value\": \"$SSH_CLIENT\", \"short\": true } ], \"color\": \"#F35A00\" } ]"

curl -s -X POST --data-urlencode "payload={\"channel\": \"$channel\", \"mrkdwn\": true, \"username\": \"ssh-bot\", $content, \"icon_emoji\": \":computer:\"}" $url

• Make the above script executable

chmod +x /var/ops/notify.sh

• In the /etc/ssh/sshd_config file, at the end of the script, add an entry like

ForceCommand "/var/ops/notify.sh"

ForceCommand runs a command when ssh allows a login. You can do many things with the ForceCommand for e.g. to ssh into another node automatically. We will post another article if you want to get into what ForceCommand can do for you.

Voila!! That’s it. Now everytime someone logs into your node, you will get an alert on Slack.

Thanks to Keerthiraja (@keet) for the above setup.