CloudOps At Scale: Secure access patterns using Bastion Host and Transfer.Sh
In the Cloud world, security is paramount. Many cloud providers are still coming up with new features to make the infra secure. In this post, Here we are going to setup a bastion server to secure the SSH access on any Cloud infra like AWS, GCP, and Azure.
Is Bastion same as Jump host?
Actually not. Jump hosts are living in the same network as you want to connect. But the Bastion host is located somewhere else and need additional security options to access other networks.
At Searce, we manage cloud infrastructure for multiple customers across AWS, GCP and Azure on 24*7 basis. When have to provide access to our CloudOps teams — it’s not efficient making writing multiple firewall rules / security groups, hop on to customer’s bastion host etc. So we have deployed our own bastion Server to access all the customer's infra.
What’s new here?
Its just bastion host setup, why we need to blog about it?
There are so many blogs and documentation about setting up a bastion server. But here we have added few ingredients to the recipe.
- SSH session recording
- 2Factor authentication enabled.
- Transfer.sh for sharing the files between any servers.
- Automated user management.
- Easy to restore the users on the new bastion server.
Enable SSH Session Recording:
This option will track every command that an user executed along the command results. With the help of this, we can easily find out if someone did something wrong. This is already published in AWS Blog, but we have done this on GCP.
Create the directory for session recording log files.
Here out admin user is sqladmin. So make this user to own this bastion log directory.
chown sqladmin:sqladmin /var/log/bastion
chmod -R 770 /var/log/bastion
setfacl -Rdm other:0 /var/log/bastion
Make OpenSSH execute a custom script on logins.
echo -e "\nForceCommand /usr/bin/bastion/shell" >> /etc/ssh/sshd_config
Remove some features from SSH and make it more restricted.
awk '!/AllowTcpForwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config
awk '!/X11Forwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config
echo "AllowTcpForwarding no" >> /etc/ssh/sshd_config
echo "X11Forwarding no" >> /etc/ssh/sshd_config
Create a shell script to capture the user's session.
cat > /usr/bin/bastion/shell << 'EOF'
if [[ -z $SSH_ORIGINAL_COMMAND ]]; then
LOG_FILE="`date --date="today" "+%Y-%m-%d_%H-%M-%S"`_`whoami`"
echo "NOTE: This SSH session will be recorded"
echo "AUDIT KEY: $LOG_FILE"
SUFFIX=`mktemp -u _XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX`
script -qf --timing=$LOG_DIR$LOG_FILE$SUFFIX.time $LOG_DIR$LOG_FILE$SUFFIX.data --command=/bin/bash
echo "This bastion supports interactive sessions only. Do not supply a command"
chmod a+x /usr/bin/bastion/shell
chown root:sqladmin /usr/bin/script
chmod g+s /usr/bin/script
Prevent bastion host users from viewing processes owned by other users, because the log file name is one of the
scipt execution parameters.
mount -o remount,rw,hidepid=2 /proc
awk '!/proc/' /etc/fstab > temp && mv temp /etc/fstab
echo "proc /proc proc defaults,hidepid=2 0 0" >> /etc/fstab
Restart the SSH service to apply /etc/ssh/sshd_config modifications.
service sshd restart
You can sync the logs files which are in
/var/log/bastion/* to S3, GCS or Azure Blob.
To verify this session recording, we can login to the bastion server and we’ll get the below welcome message.
Also we can find the log files.
[root@XXXXXXX bhuvaneshwaran_r]# ls /var/log/bastion/
Configure SMTP with Postfix and integrate AWS SES:
Before installing google authenticator and other things, we are going to install postfix mail agent and integrate with AWS SES. (if you have any own mail servers then you can ignore this part)
yum remove sendmail
yum install postfix
yum install cyrus-sasl-plain
alternatives --set mta /usr/sbin/postfix
#Replace email-smtp.us-west-2.amazonaws.com with your SES region or SMTP server.
smtp_tls_note_starttls_offer = yes
smtp_tls_security_level = encrypt
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
relayhost = [email-smtp.us-west-2.amazonaws.com]:587
smtp_sasl_auth_enable = yes
smtp_use_tls = yes
smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt
sudo postmap hash:/etc/postfix/sasl_passwd
sudo chown root:root /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
sudo chmod 0600 /etc/postfix/sasl_passwd /etc/postfix/sasl_passwd.db
sudo postconf -e 'smtp_tls_CAfile = /etc/ssl/certs/ca-bundle.crt'
sudo postfix start; sudo postfix reload
Send the test message:
echo "test mail from postfix" | mailx -r "email@example.com (Bastion Admin)" -s "Bastion host test" firstname.lastname@example.org
Enable 2FA with Google Authenticator:
Install Google authenticator:
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install google-authenticator
Setup Google Authentication in
#remove this line or comment it.
auth substack password-auth
#add this line in the end
auth required pam_google_authenticator.so
#Please take a copy of the sshd_config file
#Use custom SSH Port
#Disable root login
#Enable Key based authendication and disable password authendication
#My admin user is sqladmin, so I don't want to use MFA for this user, so if Google Authendicator is not working, I can login with this admin user and fix it. So exclude sqladmin user from MFA.
Match User "sqladmin"
Match User "*,!sqladmin"
Save the config file and restart it.
service sshd restart
Automate User Creation:
This shell script will help to create the user with google authenticator along with the Publickey authentication. Then automatically then send the private key and Google authenticator’s QR code in email.
Also it’ll sync the user’s public/private keys and google authenticator files to GCS bucket. (you can modify this script to sync this information to S3 or Azure blob.
- Username: Name of the user
- Email: email address to send the private key and google QR code.
- Super User: Does this user need root access? If yes
Create an user and test the MFA:
User Name: bhuvanesh
Email Address: email@example.com
Do you want Super User access [Y or N]: Y
Configure Transfer.sh on bastion:
Now we are going to setting up transfer.sh via docker.
yum install docker
service docker start
docker pull dutchcoders/transfer.sh:latest
I have a 100G volume and attached it to
/transfer and I want all the uploaded files should go to this mount point.
- tmp dir:
- upload files location:
docker run -d --restart always --privileged=true -i -v "/transfer/uploads:/uploads" -v "/transfer/tmp:/tmp" --publish 80:8080 dutchcoders/transfer.sh:latest --provider local --basedir /uploads --temp-path=/tmp
Open your browser and hit the IP of the bastion server.
You can use the custom port, SSL and etc in this Docker container. From our setup this port is only opened to customer’s jump host. So we’ll transfer the files from our computers to the bastion server and download it from customer’s jump box.
This is our current setup, but you can do a lot of customizations. This bastion server only helps you to switch to Linux servers, not windows. Use custom port for bastion server’s SSH or restrict SSH port from the security group. You can take advantage of login page for transfer.sh web interface and many things.
We used the below lines inside the user creation shell script to transfer files in one command.
#Setup Transfer.sh alias
curl --progress-bar --upload-file "$1" http://10.10.10.10/$(basename $1) | tee /dev/null;