Logging Bash History

For Red Teams, Penetration Testers, and Security Researchers

Bash is the command line interface so many of us use daily during our operations. Yet, how often are you at a loss of remembering what command was run during that thing that you worked so hard to achieve? Or how easily can you retrieve the time and date you executed a nmap scan against your target? The Red Team community, as of late, has been apt to apply systematic logging across their infrastructure to keep track of their Command and Control usage, and folks much more clever than I have gone above and beyond to automate alerting based on logs from their Redirectors and C2 agents through the RedELK project. In this post, I add Bash History to the logging movement and how I’ve implemented a standard logging format for my team’s ELK stack.

Why Log Bash History?

Bash is shipped as the default shell for many Linux systems and is full of functionality, yet its default logging is very limited. Logging Bash History can add context to our operational timeline or enhance it further when running scripts or programs that do not fall within our Command and Control frameworks, which typically have very robust logging capabilities. The greatest point I would like to make is that Bash is likely going to be the shell that you use when standing up disposable infrastructure. Having commands run in Bash across all of your infrastructure logged to one central location is an incredible capability to have, especially when your infrastructure may have a variety of functions.

Chuck Norris knows how to control loggers

TL;DR

In this post, I use a variety of technologies to configure logging for the Bash shell and forward those logs to an Elasticsearch, Logstash, and Kibana (ELK) Stack built in Docker. This makes it easy to demonstrate how to configure the various components to get Bash logging set up and should be simple to replicate in your own infrastructure. If you want to jump straight in on your own, you can head over to my demonstration repository. You’ll also find a helpful Ansible playbook to get your infrastructure up to speed immediately:

https://github.com/cyberbutler/bash-logging-elk

This repository can be used to get up and running with all the necessary configurations immediately and includes a Bash Operator container running an Ubuntu Docker image to demonstrate logging to the ELK stack easily.

Configuring the Operator’s Bash Environment

To set up bash logging, we will be using the PROMPT_COMMAND environment variable. The contents of PROMPT_COMMAND will be executed before a Bash prompt is returned (meaning after every interactive command is executed in a terminal). This Bash feature will allow us to send logs through a Local Facility and write them to a log file using rsyslog. Finally, we will use filebeat to forward the logs to an ELK stack that we will build using Docker!

Woof, that is a lot I know, but it’s not too complicated. Here is a fancy diagram of what will happen:

Setup

For this project, I used an Ubuntu Docker container and depending on your Operating System, some of the commands may differ to get the necessary packages installed properly. If you are interested in using my containers, see my Bash Logging ELK Stack repository.

Ensure you have rsyslog installed:

apt-get install -y update && apt-get install -y rsyslog

We will also need to get filebeat installed. In order for the elastic products to work together, it’s critical that all of the components are running the same version. For this entire configuration, I used version 7.6.2.

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.2-amd64.deb
dpkg -i filebeat-7.6.2-amd64.deb

PROMPT_COMMAND

Next, let’s get the PROMPT_COMMAND environment variable persistently configured. We can do this by adding the following command to /etc/bash.bashrc. Each new bash prompt will source directly from this file, ensuring the environment variable is loaded.

/etc/bash.bashrc

export PROMPT_COMMAND=’RETRN_VAL=$?; if [ -f /tmp/lastoutput.tmp ]; then LAST_OUTPUT=$(cat /tmp/lastoutput.tmp); rm /tmp/lastoutput.tmp; fi; logger -S 10000 -p local6.debug “{\”user\”: \”$(whoami)\”, \”path\”: \”$(pwd)\”, \”pid\”: \”$$\”, \”b64_command\”: \”$(history 1 | sed “s/^[ ]*[0–9]\+[ ]*//” | base64 -w0 )\”, \”status\”: \”$RETRN_VAL\”, \”b64_output\”: \”$LAST_OUTPUT\”}”; unset LAST_OUTPUT;

Ok so there is a lot going on here, lets extract and clean up this script a bit so we can break it down:

Line 1 sets a variable that contains the return value of the previously run command. As PROMPT_COMMAND runs before the redisplay of our prompt, it ultimately means that the script will execute after each command we run.

Lines 2–5 check for the existence of a file called /tmp/lastoutput.tmp, sets the value of LAST_OUTPUT to the contents of the file, and removes the file if it exists. This file will store the output of our command through a utility function we will build later.

Line 7 uses the logger command with a max of 10000 characters and sets the priority to the local facility level local6.debug. This will allow us to later capture the log using rsyslog.

Lines 8–13 contain the log message. In our case, we are using JSON format. This will make our lives a lot easier when it comes time to parse the log in Logstash during our ELK stack build. As you should see we are logging 6 fields, including:

• Current User
• Current Path
• PID
• The previous run command extracted using history and base64 encoded. Base64 encoding ensures that the command is properly escaped in our JSON object.
• The return value of the previous run command
• The base64 encoded output of the command (if it exists). We will dive deeper into this field later.

Finally, on Line 15, we unset the LAST_OUTPUT variable so it does not persist between commands.

LAST_OUTPUT

Now we can set up our helper function to capture the output of our commands so that we can push them to the ELK stack. Because there is no built-in way to automatically capture the output of a command, we must either run the command twice or use a helper function to capture the output in the PROMPT_COMMAND script. As you can imagine, the prior is not the best option when running commands that may interact with a target or make a configuration change. The following command should be included in your /etc/bash.bashrc file.

logoutput() {
  output=$(while read input; do echo “$input”; done < “${1:-/ dev/stdin}”);
  echo -e “$output\n”;
  echo -e “$output” | head -c 10000 | base64 -w0 > /tmp/lastoutput.tmp;
  return $?;
}

As shown, the logoutput command will capture the output of a command or read the contents of a file, base64 encode the first 10,000 characters, and write it to /tmp/lastoutput.tmp (remember this file was used in our PROMPT_COMMAND earlier!). This helper function can be used against a file on the system or can have STDOUT piped to it for easy logging! You can use it as follows:

echo “It works!” | logoutput

That concludes the modifications we must make to /etc/bash.bashrc and we can now move on to setting up rsyslog. You should ensure that the following is appended to your /etc/bash.bashrc configuration:

export PROMPT_COMMAND=’RETRN_VAL=$?; if [ -f /tmp/lastoutput.tmp ]; then LAST_OUTPUT=$(cat /tmp/lastoutput.tmp); rm /tmp/lastoutput.tmp; fi; logger -S 10000 -p local6.debug “{\”user\”: \”$(whoami)\”, \”path\”: \”$(pwd)\”, \”pid\”: \”$$\”, \”b64_command\”: \”$(history 1 | sed “s/^[ ]*[0–9]\+[ ]*//” | base64 -w0 )\”, \”status\”: \”$RETRN_VAL\”, \”b64_output\”: \”$LAST_OUTPUT\”}”; unset LAST_OUTPUT; ‘
logoutput() { output=$(while read input; do echo “$input”; done < “${1:-/dev/stdin}”); echo -e “$output“; echo -e “$output” | head -c 10000 | base64 -w0 > /tmp/lastoutput.tmp; return $?; }

Write to bash.log

Add the following for rsyslog to monitor the local facility local6 in the following file: /etc/rsyslog.d/bash.conf

local6.* /var/log/bash.log

Restart rsyslog:

service rsyslog restart

Next, we will use logrotate to ensure that the logs do not get too big by creating new log files. In /etc/logrotate.d/rsyslog we must ensure that the path to our bash logs, /var/log/bash.log is included with parameters. To do this quickly, run the following command to handle bash.log rotation similar to how /var/log/messages is handled:

sed -i ‘/^\/var\/log\/messages/a /var/log/bash.log’ /etc/logrotate.d/rsyslog

The above sed command allows us to append /var/log/bash.log to the line after /var/log/messages in /etc/logrotate.d/rsyslog. This should result in a /etc/logrotate.d/rsyslog file similar to the following:

…snip…
/var/log/messages
/var/log/bash.log
{
  rotate 4
  weekly
  missingok
  notifempty
  compress
  delaycompress
  sharedscripts
  postrotate
  /usr/lib/rsyslog/rsyslog-rotate
  endscript
}

Test it out!

Take a moment to run a few commands! See your handy work by tailing /var/log/bash.log and running a few commands!

You should have a log that looks like this:

Jul 23 18:57:08 ef87d5442553 root: {"user": "root", "path": "/var/log", "pid": "221", "b64_command": "ZWNobyAnOiknCg==", "status": "0", "b64_output": ""}

Build the ELK Stack with Docker

Awesome, we are almost done configuring our bash environment. All that is left is to set up Filebeat to forward the bash logs to our ELK stack, however, to make that quicker and easier, let’s go ahead and set up our ELK stack first. ELK will be our central log location.

I wanted to take a moment to call out this handy ELK stack build tool called RedELK. RedELK is a great platform for Red Teams to enrich operational logs from their redirectors and C2 tooling. For the purposes of this post, however, we will be building our own ELK stack. I would have liked to use RedELK to demonstrate bash logging, but its current deployment process can be cumbersome and the instructions I have laid out in this post can be easily replicated on your own RedELK deployment!

With that said, to configure our ELK Stack, we are going to take advantage of Anthony Lapenna’s Docker Compose repository for ELK and customize it to fit our needs.

git clone https://github.com/deviantony/docker-elk
cd docker-elk

Taking a look at the docker-compose.yml file, you can see three separate services that are required to get up and running with a shared network so they can communicate: ElasticSearch, Logstash, and Kibana. You should take a moment to change some of the parameters to your liking. You may change your logon credentials or modify how much RAM is allocated to the ElasticSearch service.

You must open port 5044 in the logstash service:

logstash:
  build:
    context: logstash/
  args:
    ELK_VERSION: $ELK_VERSION
  volumes:
  - type: bind
    source: ./logstash/config/logstash.yml
    target: /usr/share/logstash/config/logstash.yml
    read_only: true
  - type: bind
    source: ./logstash/pipeline
    target: /usr/share/logstash/pipeline
    read_only: true
  ports:
    - "5044:5044/tcp"
    - "5000:5000/tcp"
    - "5000:5000/udp"
    - "9600:9600"

Parsing logs with Logstash

Drop the following into ./logstash/pipeline/logstash.conf(if you changed your credentials to Elasticsearch, you will need to do so as well in this file):

Let’s do a quick breakdown of the above:

Lines 1–5 set the input port and expect it to be a filebeat connection.

Line 7 starts the filter declaration, this block is where we parse our logs.

Line 8 will match only on our “bash” logs. This allows us to add other filters on different types of logs if we choose to add more in the future.

Lines 9–11 uses a grok filter which allows us to do the actual parsing. This filter uses regular expressions and will extract fields. Most notably, the json_message field will be parsed in another filter soon.

Lines 13–15 parse the syslog_timestamp field and set it as the timestamp for elasticsearch to index on. This will make it so that we can sort more accurately on when the command was run, instead of when the log was received by Logstash. This, however, means that we rely on the time of our operational systems to match the time zone of our ELK stack.

Lines 16–18 parse out the json_message and assign each key as its field. Hopefully you see why we went through the trouble to put our log format in JSON now :)

Lines 19–26 is where some of the magic happens. Here we run Ruby code which extracts the contents of b64_command and b64_output, decodes them, and assigns them to the new fields command and output respectively.

Lines 31 and onward set the destination of the logs, which is our elasticsearch container. These logs are saved in an index that will be in the format bash-%{+YYYY.MM.dd}.

Start the ELK Stack

Getting up and running is as simple as running the following docker-compose command:

docker-compose up -d

Filebeat

To wrap up this build, move back to your bash environment and create the following file in /etc/filebeat/filebeat.yml. You should replace the output.logstash hosts variable to point to your Logstash server (or leave as is if you are using my docker-compose containers).

Wrapping it all up

Wow, we are almost done, and what an adventure. To wrap everything up, go to your bash prompt and fire off a few commands, such as ls, whoami, echo 'I did it! :)' | logoutput. This will get some logs forwarded to Logstash and indexed in Elasticsearch. Once that is done, you can log into Kibana by visiting the host you set your ELK stack up on, for me it was http://localhost:5601/. You should be prompted with the following (if you chose not to change your credentials, the default login is elastic:changeme)

You should then navigate to /app/kibana#/management/kibana/index_patterns so that we can register our new bash index:

Click on “Create index pattern” and type in bash-*:

Click “Next step” and select @timestamp as the Time Filter field name. Then click “Create index pattern”

Navigate to /app/kibana#/discover and you should see your bash logs!

Add a couple columns and customize how you wish! This is how I like to save my fieldset:

Conclusion

Logging Bash commands to a single source like an ELK stack is extremely handy when you are running an OP. I cannot tell you how many times we’ve run into a case where being able to hunt down a command we ran from something months ago made our lives so much better. Distributing this configuration to all of our cloud redirectors, local operational boxes, Teamservers, and the like makes it incredibly easy to keep track of the work that is being done. If you are like me and live out of a terminal, logging Bash commands is an absolute must.

Happy Logging!

Website: www.maveris.com

Email: info@maveris.com

Maveris exists to help your organization reach its fullest potential by providing thought leadership in IT and cyber so you can connect fearlessly. To learn more visit us at: www.maveris.com