Create GeoIP dashboards in Grafana from iptables logs

How to add geographical information to your firewall logs and display them in Grafana

Sample GeoIP Dashboard in Grafana

Sometimes when you host your own server you might want to see where requests are coming from. There are many solutions on how to map IP addresses to geographical locations, with web-server logs for example. There are also many options to display them with various Monitoring solutions. However, if you have multiple services which create logs, it might get tedious to configure all of them to map access logs to GeoIP. But when you are able to take the IP addresses directly from your firewall, you get all information required from one place.

In this post I will explain how I configured Grafana to display GeoIP information coming from iptables logs.

What you will need

• A linux machine with iptables and docker installed (you can do it without docker, but in this post I will be using docker for it)
• A Grafana stack including Promtail and Loki
• Syslog-ng to enrich the iptables logs with GeoIP information
• Some actual people (or bots) accessing your server

NOTE: As geoip is available since February 23 in promtail’s pipeline_stages we no longer need to use Syslog-ng. Please read at the end of this post for implied changes.

Getting started

First we will install iptables on the system (assuming a Debian based operating system):

apt install -y iptables docker docker-compose

Note that the docker instructions might differ, please see the official documentation.

Setting up the Grafana stack

To get started easily, create a docker-compose.yml in a directory called grafana-stack (for example) with the following content (make sure to replace all placeholders in <>):

services:

  grafana:
    image: grafana/grafana:9.2.3
    hostname: grafana
    container_name: grafana
    restart: unless-stopped
    environment:
      GF_SECURITY_ADMIN_USER: <PLACE_YOUR_ADMIN_USERNAME_HERE>
      GF_SECURITY_ADMIN_PASSWORD: <PLACE_YOUR_ADMIN_PASSWORD_HERE>
      GF_SERVER_DOMAIN: localhost
    ports:
      - '127.0.0.1:3000:3000'
    networks:
      - grafana

  loki:
    image: grafana/loki:2.7.1
    hostname: loki
    container_name: loki
    restart: unless-stopped
    networks:
      - loki
      - grafana

  promtail:
    image: grafana/promtail:2.7.1
    hostname: promtail
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./promtail.yaml:/etc/promtail/promtail.yaml:ro
    command: 
      - '-config.file=/etc/promtail/promtail.yaml'
    networks:
      - loki
      - promtail

networks:
  grafana: {}
  loki: {}
  promtail:
    name: promtail

While Grafana and Loki should work out of the box with this configuration (please do adjust the configuration for a production setup), we need to configure Promtail to listen on a port for syslog logs (as of the time of writing, Promtail has no option to add GeoIP information directly).

Thus, we create a file called promtail.yaml on the same level as the docker-compose with the following content:

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  log_level: info

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

# This is the endpoint we will be sending our enriched iptables logs to
- job_name: iptables
  syslog:
    listen_address: 0.0.0.0:8514
    labels:
      job: iptables

The line below makes sure that this configuration gets mounted into the container running Promtail (it is already present in the docker-compose above):

volumes:
      - ./promtail.yaml:/etc/promtail/promtail.yaml:ro

To get everything running, run docker-compose up -d. Grafana should then become accessible on http://localhost:3000. What we still need to do is to connect Loki to it. To achieve this, we need to visit Grafana in our browser and add Loki as a datasource. As we use docker, we need to use http://loki:3100 as address. All the required information to achieve this, is written in the documentation just below.

Loki in Grafana | Grafana Loki documentation

You might have noticed that there is no persistence for neither Loki and Grafana. If we wanted to save the data of them, we would also have to add the correct volume mounts to the compose file.

Configure iptables to log fresh IP addresses

First, we will create a custom chain where we will add our logging rules to:

iptables -N IP_LOGGING

Then we make sure that we do not log connections coming from private IP ranges:

iptables -A IP_LOGGING -s 10.0.0.0/8,172.16.0.0/12,192.168.178.0/24,127.0.0.1/32 -j RETURN

As many botnets and scanners create a huge amount of noise in the internet, we want to make sure that we will not end up logging each connection to our host. Thus we will make use of the recent extension in iptables. First, we add a rule which checks a list named iptrack, containing a list of connections. If the currently handled source IP address has been seen already in the past 12 hours (43200 seconds) it will be present in this list and we want to ignore it:

iptables -A IP_LOGGING -m recent --rcheck --seconds 43200 --name iptrack --rsource -j RETURN

If the IP address is known already, the jump target will be applied and we go out of our custom chain. If it is an unknown address, we want to add it to the list:

iptables -A IP_LOGGING -m recent --set --name iptrack --rsource

And finally log it to syslog on the host machine:

iptables -A IP_LOGGING -j LOG --log-prefix "[iptables] "

Thanks to the — log-prefix flag, each line logged will contain a uniquely identifiable string ([iptables] ) which we will use later.

Up until now, no IP address will actually be logged, since we added all rules into a custom chain. If we now want to log those, we need to add it to the desired chains which a connection will be going through. For host connections this will be the INPUT chain, and for forwarded connections (to other hosts, or to docker containers for example) this will be the FORWARD chain. If you have docker installed, I recommend adding the custom chain to the chain called DOCKER-USER instead of FORWARD , which in our case will be better, as we are using docker for the setup anyway (the DOCKER-USER chain is inserted in the FORWARD chain by docker, see more in the official documentation):

iptables -A INPUT -j IP_LOGGING
iptables -A DOCKER-USER -j IP_LOGGING

Out iptables rules should look like this:

# some other stuff above here
-A INPUT -j IP_LOGGING
-A DOCKER-USER -j IP_LOGGING
-A IP_LOGGING -s 10.0.0.0/8 -j RETURN
-A IP_LOGGING -s 172.16.0.0/12 -j RETURN
-A IP_LOGGING -s 192.168.178.0/24 -j RETURN
-A IP_LOGGING -s 127.0.0.1/32 -j RETURN
-A IP_LOGGING -m recent --rcheck --seconds 43200 --reap --name iptrack --mask 255.255.255.255 --rsource -j RETURN
-A IP_LOGGING -m recent --set --name iptrack --mask 255.255.255.255 --rsource
-A IP_LOGGING -j LOG --log-prefix "[iptables] "

This snipped is part of an export of iptables-save, that comes with the iptables-persistentpackage. You can use this command when you want to preserve the firewall rules between system reboots.

If now someone is accessing your server, iptables will log all new connections to /var/log/syslogin a format similar to this:

Jan 13 18:06:45 hostname kernel: [79240.520946] [iptables] IN=eno1 OUT=grafana-br0 MAC=dest-mac:source-mac:frame-type SRC=source-ip DST=dest-ip LEN=60 TOS=0x00 PREC=0x00 TTL=119 ID=0 DF PROTO=TCP SPT=source-port DPT=destination-port WINDOW=65535 RES=0x00 ACK SYN URGP=0 

But for this setup we want to redirect log lines coming from iptables to a separate file, and we can achieve this by creating a file in/etc/rsyslog.d . We will name this file 01-iptables.conf:

Note that you do not necessarily need to store the messages in a file, you can also send them to syslog-ng directly. At the end of this post is a short summary on how to achieve this. However, I personally like to store this information in a dedicated file.

# Write all logs coming from iptables to a separate file
:msg,contains,"[iptables] " /var/log/iptables.log 

Now to apply the settings we need to restart rsyslog, run systemctl restart rsyslog.service and all iptables logs will now be redirected to /var/log/iptables.log .

Enriching the logs with GeoIP data

To add GeoIP data to the logs, we will make use of syslog-ng and maxmind’s free GeoIP database, please check the link, create an account and get your license key.

We will again use docker to enrich the data. Create another directory called geoip-enrichment (for example). In there create another docker-compose.yml with the following content (again make sure to replace all placeholders in <>):

services:

  syslog:
    image: balabit/syslog-ng:4.0.1
    container_name: syslog
    hostname: syslog
    restart: unless-stopped
    command:
      - '--no-caps'
    volumes:
      - ./syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf:ro
      - /var/log/iptables.log:/var/log/iptables.log:ro
      - geoip:/usr/share/GeoIP
    depends_on:
      geoip:
        condition: service_completed_successfully
    networks:
      - promtail

  geoip:
    image: maxmindinc/geoipupdate
    container_name: geoip
    restart: on-failure
    environment:
      GEOIPUPDATE_ACCOUNT_ID: <YOUR_GEOIP_ACCOUNT_ID>
      GEOIPUPDATE_LICENSE_KEY: <YOUR_GEOIP_LICENSE_KEY>
      GEOIPUPDATE_EDITION_IDS: GeoLite2-City
    volumes:
      - geoip:/usr/share/GeoIP

volumes:
  geoip: {}

networks:
  promtail:
     external: true

Note the mounted volumes, which mount our custom iptables log into the syslog container. And again an additional configuration, this time for syslog-ng. We also added the promtail network as external, so that syslog-ng is able to ingest the enriched logs into promtail.

Next, create a file called syslog-ng.conf on the same level as the docker-compose:

@version: 4.0

source s_ipt {
  # Load the file mounted from the host into this container
  file(
    "/var/log/iptables.log",
    follow-freq(1)
  );
};

parser p_ipt_kv {
  # Add an ipt. prefix to all keys which have been logged by iptables
  kv-parser(prefix("ipt."));
};

parser p_ipt_geoip {
  # Enrich the logs from iptables with the data stored in the database from maxmind based on the source IP field
  # This step will add new attributes to the logs, like e.g. geoip.location.longitude and geoip.location.latitude
  geoip2(
    "${ipt.SRC}", 
    prefix("geoip.") database("/usr/share/GeoIP/GeoLite2-City.mmdb")
  );
};

template t_ipt_json {
  # This template converts the plaintext syslog to json and removes some keys from the log which are unimportant for our use-case
  # feel free to customize based on your needs
  template(
    "$(format-json --scope nv-pairs --key ISODATE --exclude ipt.WINDOW --exclude ipt.TTL --exclude ipt.RES --exclude ipt.TOS --exclude ipt.ID --exclude ipt.MAC --exclude ipt.PREC --exclude ipt.URGP --exclude ipt.LEN --exclude MESSAGE --exclude SOURCE --exclude LEGACY_MSGHDR --exclude HOST --exclude FILE_NAME --exclude PROGRAM --exclude HOST_FROM)"
  );
};

destination d_ipt_promtail {
  # Send the enriched and filtered data to promtail
  # Note that the hostname `promtail` only works in the docker setup, if it is a remote host, adjust accordingly
  tcp(
    "promtail" port(8514) flags(syslog-protocol) 
    template(t_ipt_json)
  );
};

log {
  # Here we source the custom log file and pass all stages to enrich the data and finally send it to promtail's syslog receiver
  source(s_ipt);
  parser(p_ipt_kv);
  parser(p_ipt_geoip);
  destination(d_ipt_promtail);
};

When you now run docker-compose up -d , docker will run the container named geoip first, which uses your credentials to download the GeoIP database from maxmind. Once this step is done, syslog-ng starts and continuously reads, enriches and transforms the data read from the mounted iptables log to finally send it to Promtail. We could now add pipelines to Promtail to further manipulate the logs, or to calculate custom metrics for Prometheus. If you are interested to do so please continue reading the Promtail documentation:

Pipelines | Grafana Loki documentation

The log lines being sent to Promtail will be looking similar to the one below:

 
{"ipt":{"SRC":"175.194.185.246","SPT":"53913","PROTO":"TCP","OUT":"gitlab-br0","IN":"eno1","DST":"172.24.0.4","DPT":"22"},"geoip":{"subdivisions":{"0":{"names":{"en":"Gyeonggi-do"},"iso_code":"41","geoname_id":"1841610"}},"registered_country":{"names":{"en":"South Korea"},"iso_code":"KR","geoname_id":"1835841"},"postal":{"code":"104"},"location":{"time_zone":"Asia/Seoul","longitude":"126.832400","latitude":"37.661000","accuracy_radius":"5"},"country":{"names":{"en":"South Korea"},"iso_code":"KR","geoname_id":"1835841"},"continent":{"names":{"en":"Asia"},"geoname_id":"6255147","code":"AS"},"city":{"names":{"en":"Goyang-si"},"geoname_id":"1842485"}},"ISODATE":"2023-01-13T18:54:22+00:00"}

We can see that we now have information about the connection in the attribute ipt and the corresponding geographical information in geoip .

Querying the data in Grafana

We should now be able to see the enriched logs in Grafana already. The following query should fetch the logs ingested into Loki through Promtail and parse the json attributes (test it on http://localhost:3000/explore):

{job="iptables"} | json

We could now filter for incoming ssh traffic like this, for example:

{job="iptables"} | json | ipt_DPT == "22"

Grafana sample dashboard

A sample dashboard making use of the ingested data is available on grafana.com:

IPTables GeoIP | Grafana Labs

Notes

You might want to set up the logging of IP addresses in a different manner, as for now we have stored the logs is a file which gets picked up by syslog-ng in the container. But you could also let syslog-ng listen on a port and send the logs directly into the container via network by configuring rsyslog accordingly. Or you could install syslog-ng natively on your system and start filtering the syslogs for iptables logs.

For the network case, just change the following few things:

• 01-iptables.conf becomes:

:msg,contains,"[iptables] " action(type="omfwd" target="127.0.0.1" port="8514" protocol="tcp")

• syslog-ng.conf, change the source s_ipt to the following:

source s_ipt {
  network(
    ip(0.0.0.0),
    port(8514),
    transport(tcp),
  );
};

• syslog’s docker-compose:

1. Remove the volume mount of the iptables log file
2. Add a new port mapping to syslog:

syslog:
  # ...
  volumes:
    # ->> remove: 
    # - /var/log/iptables.log:/var/log/iptables.log
  ports:
    - "127.0.0.1:8514:8514"

For the other case (running a natively installed syslog-ng instance), just add a filter to syslog-ng’s configuration (which in this case might reside in /etc/syslog-ng/conf.d:

# sourcing system logs

filter f_ipt { 
  message("iptables");
};

# lots of other stuff

log {
  source(s_src);
  filter(f_ipt);
  parser(p_ipt_kv);
  parser(p_ipt_geoip);
  destination(d_ipt_promtail);
};

You might also like to have logs from a dedicated firewall device, this is also possible with similar configuration. Just make sure your syslog-ng container opens the network port for systems which want to send iptables logs via the network and adjust the rsylog configuration accordingly (replace the target address).

Promtail and GeoIP

Since February 2023 promtail does also support enriching logs with GeoIP data, which means that we no longer need to use a Syslog-ng instance in between.

To do so, first mount the geoip volume to the promtail container like we did with the Syslog-ng container. Then adjust promtail’s iptables job in promtail.yaml and add pipeline stages to enrich the log lines received via rsyslog (Syslog-ng) with the mounted database:

scrape_configs:

 - job_name: iptables
    syslog:
      listen_address: 0.0.0.0:8514
      labels:
        job: iptables
    pipeline_stages:
      - logfmt:
          mapping:
            input_interface: IN
            output_interface: OUT
            src_ip: SRC
            protocol: PROTO
            dst_port: DPT
      - geoip:
          db: /usr/share/GeoIP/GeoLite2-City.mmdb
          db_type: city
          source: src_ip
      - labeldrop:
          - geoip_timezone
          - geoip_subdivision_name
          - geoip_subdivision_code
          - geoip_postal_code
      - labels:
          input_interface:
          output_interface:
          protocol:
          dst_port:
      - output:
          source: message

With this configuration, the following stages are run:

• logfmt: Parse the log lines in with logfmt and add extracted data (key value pairs to use in subsequent stages) like input_interface, output_interface, src_ip and so on.
• geoip: Use the extracted field `src_ip` to add the GeoIP labels described in the geoip stage documentation.
• labeldrop: Remove unwanted lables which geoip added to the log line.
• labels: Add more custom labels to the log line. Here we use the ones which we extracted in the logfmt stage except the IP address since these are not fixed and would result in a high cardinality.
• output: Forward the original message with the new labels to loki.

Now we can eliminate the Syslog-ng container and send the iptables logs directly to promtail by adjusting the rsyslog configuration.
01-iptables.conf now becomes:

:msg,contains,"[iptables] " action(type="omfwd" target="127.0.0.1" port="8514" protocol="tcp" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted")

NOTE: Make sure that you adjust the docker-compose of promtail and add the port binding 127.0.0.1:8514:8514 before restarting rsyslog.

Now restart the promtail container and rsyslog and you should see the new labels being added to the log lines in grafana. This now also simplifies the dashboards which we created. The new revision is uploaded to grafana.com.