Hosting your own CTF

This article is about my experiences in setting up the infrastructure for S.H.E.L.L. CTF 2021. In this blog, we’ll cover :

1. Setting up the CTFd platform on your instance.
2. Containerizing CTFd and some challenges, with docker.
3. Setting up Nginx for rate-limiting to prevent brute-force/DDOS.
4. Setting up Cloudflare as a front, for DNS caching and logging of requests.
5. Containerizing challenges and using xinetd to grant simultaneous access of some challenges(eg pwn or python-listener etc.) to multiple users.

The CTFd Platform

CTFd/CTFd

CTFd is an easy-to-use, open-source, CTF hosting platform. It comes with everything one might need to host a CTF. Some features include:
- An admin panel to configure the environment,
- Add and edit challenges (dynamic scores and add hints, etc.),
- View statistics, and
- Modify the home page.

Deploying CTFd

There are 3 ways in which CTFd can be deployed on your server :

1. Clone the repository, install the requirements via pip, configure it to your liking, and use python serve.py or flask run in a terminal to drop into debug mode.
2. One can use Docker Compose with the following command from the source repository:docker-compose up which will use the docker-compose.yml file.
3. One can also use the auto-generated Docker image with the following command:docker run -p 8000:8000 -it ctfd/ctfd

I used option 3 as it was the easiest to set up and worked without hassles.

We start by installing docker to our instance. Since we used Ubuntu 20.04, I followed the steps mentioned here: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04. The steps are :

sudo apt update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
sudo apt update
sudo apt install docker-ce
sudo systemctl status docker

Once docker is installed, we can execute the docker run command docker run -p 8000:8000 -it ctfd/ctfd. This should startup CTFd running on port 8000 on your server.

Navigate to port 8000 on <YOUR_SERVER_IP>:8000 or if you have linked a domain to your server, then at <your domain>:8000 on a browser.

After completing the initial setup for CTFd, you should be able to now access the Admin Panel to manage your CTF!

Firewall and Nginx for rate limiting

Nginx is a reverse proxy server, i.e. it accepts incoming connections to its port and then routes them to another service running on another port on the machine. We will be setting up Nginx and configuring it to do the following things:

1. Previously, you accessed your server using <your domain>:8000, we will instead route <your domain> to CTFd running on port 8000 automatically.
2. We’ll set up rate limiting to limit both the number of requests per second to CTFd and also the maximum number of simultaneous connections to it from a single host.
3. (optional) If you’re going to be using Cloudflare, we’ll also reconfigure Nginx to correctly log the original user’s IP address, instead of logging only Cloudflare IPs in Nginx logs.

Installing ufw (firewall) and Nginx on Ubuntu 20.04 server :

sudo apt update 
sudo apt install nginx ufw

Allowing SSH, HTTP and HTTPS through the firewall :

sudo ufw allow 'Nginx Full'
sudo ufw allow 'OpenSSH'

Now to enable the firewall :

sudo ufw enable

Now if you visit the domain, you should see the default Nginx page.

We need to route Nginx reverse proxy to port 8000, to show our CTFd page.

Create a file at /etc/nginx/sites-available/yourdomain.com (replace yourdomain.com with your domain or just your IP address if you don’t have a domain) with the following contents.

limit_req_zone  $binary_remote_addr zone=mylimit:10m rate=10r/s;
limit_conn_zone $binary_remote_addr zone=addr:10m;
server { 
    server_name yourdomain.com; 
    limit_req zone=mylimit burst=15; 
    limit_conn addr 10; 
    limit_req_status 429; 
    client_max_body_size 8M; 
    location / {      
        proxy_pass http://localhost:8000;        
        proxy_http_version 1.1;        
        proxy_set_header Upgrade $http_upgrade;        
        proxy_set_header Connection 'upgrade';        
        proxy_set_header Host $host;        
        proxy_cache_bypass $http_upgrade;  
    }
}

This sets up rate-limiting at 10 requests per second, and a max of 10 simultaneous connections per IP address at a time, and also tells Nginx to route requests to yourdomain.com at port 8000.

If you are using Cloudflare, replace $binary_remote_addr everywhere above with $http_cf_connecting_ip. This makes Nginx read the user IPs and limit requests on their basis, not on the Cloudflare server’s IP.

We create a symlink to the file you created in the previous step in /etc/nginx/sites-enabled and reload Nginx, and we’re done!

sudo ln -s /etc/nginx/sites-available/yourdomain.com /etc/nginx/sites-enabled/yourdomain.com
sudo systemctl restart nginx

Try visiting yourdomain.com and you will see the CTFd page. Play around with the rate limit and see what suits your need.

For those using Cloudflare

The default Nginx configuration will log every request in /var/log/access.log but there is a slight issue: the origin IP of each request is going to be a Cloudflare server. This way we won’t be able to trace back requests of illegitimate or malicious users to their IP, during the CTF.

We can fix this by making a small change in the http section in /etc/nginx/nginx.conf to log the real User IP instead. Replace your .conf file with this:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	# server_tokens off;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

log_format main '$http_cf_connecting_ip - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent "$http_referer" '
'"$http_user_agent"' ;
	access_log /var/log/nginx/access.log main;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

Execute sudo systemctl restart nginx and now, Nginx should log the user IP instead of Cloudflare server IPs!

Setting up https:// using certbot and LetsEncrypt

Certbot

SSL Certificates normally cost money, but we’re gonna use Let’s Encrypt, a service whose primary goal is to provide free certificates to websites on the internet.

You can refer to the amazing documentation at certbot’s official site for instructions, but here is how you can set it up on Ubuntu 20.04:

sudo apt-get update
sudo apt-get install software-properties-common
sudo add-apt-repository universe
sudo apt-get update 
sudo apt-get install certbot python3-certbot-nginx

Once you have certbot installed, run

sudo certbot --nginx

This will prompt you with a list of domains configured with Nginx, select the domain we set up with CTFd, and follow the series of configuration options.

Make sure to choose the option of always redirect to HTTPS, as we don’t want users accessing our site over HTTP.

Once you have answered all the prompts, try going to your domain, you should see it redirect to https://. Click on the tiny lock to the left of the address and your browser should say that the connection is secure.

Setting up Cloudflare

Cloudflare is great to have. With all its features your server load decreases gradually and the performance increases. It would also serve the cached version of your web pages when your web server goes offline for maintenance. The best part is that it has a free version that does a lot of the aforementioned things.

1) Log on to http://www.cloudflare.com/ and sign up for a new account. You’ll need to fill a sign-up form.

2) The next step would be to add your website by entering the URL as shown below:

3) Cloudflare will then scan your existing domain records.

This process is likely to take a few seconds. Once the scan is complete, you’ll be directed to another DNS Zone file where you can verify that all records have been successfully transferred.

Here, you can choose to keep particular subdomains on or off the Cloudflare network. An orange cloud represents that the specific subdomain will be cached and will be served through Cloudflare, while a gray cloud represents that the particular subdomain will bypass Cloudflare and all requests will go directly to the webserver.

4) Next, you can select the plan for your Cloudflare account. Remember that the basic use of Cloudflare is free. You may choose a paid plan in case you want additional features. We just went with the free plan and it worked well for our needs.

5) Lastly, Cloudflare will provide you with two DNS name servers. For example, eva.cloudflare.com. You’ll need to replace the DNS name servers on your domain name management dashboard with the ones provided by Cloudflare and wait for them to resolve.

After the wait, you will see your domain active on the Cloudflare dashboard and it will analyze and display the traffic for you.

Dockerizing challenges and setting up Xinetd for hosted challenges

It's always good to containerize the challenges that will be hosted. This ensures that if a challenge works on your local computer, it will work on the server.

Also if someone were to obtain remote code execution, either by solving the challenge or with malicious intent, they would be in a containerized environment and the server would be safe

We used it for Web and Python(Crypto) challenges. I will take the example of a Python Crypto challenge.

We require the following three files:

• Dockerfile
• ctf.xinetd
• start.sh

Dockerfile

The Dockerfile uses ubuntu:16.04 to host the challenge. We update and upgrade the distribution, install xinetd python3 and pip3 (pip install modules). Add a lower privileged user ‘ctf’, set its home directory as the work dir. We make important file system nodes inside the workdir, copy some useful binaries like sh, ls etc., and copy the ctf.xinetd and start.sh scripts to the container. Then we copy the actual script from bin directory on local machine to the container, give it appropriate permissions, execute start.sh script and expose a port.

FROM ubuntu:16.04
RUN sed -i "s/http:\/\/archive.ubuntu.com/http:\/\/mirrors.tuna.tsinghua.edu.cn/g" /etc/apt/sources.list && \
    apt-get update && apt-get -y dist-upgrade && \
    apt-get install -y xinetd python3 python3-pip
RUN useradd -m ctf
RUN pip3 install pycrypto
WORKDIR /home/ctf
RUN cp -R /lib* /home/ctf && \
    cp -R /usr/lib* /home/ctf
RUN mkdir /home/ctf/dev && \
    mknod /home/ctf/dev/null c 1 3 && \
    mknod /home/ctf/dev/zero c 1 5 && \
    mknod /home/ctf/dev/random c 1 8 && \
    mknod /home/ctf/dev/urandom c 1 9 && \
    chmod 666 /home/ctf/dev/*
RUN mkdir /home/ctf/bin && \
    cp /bin/sh /home/ctf/bin && \
    cp /bin/ls /home/ctf/bin && \
    cp /bin/cat /home/ctf/bin
COPY ./ctf.xinetd /etc/xinetd.d/ctf
COPY ./start.sh /start.sh
RUN echo "Blocked by ctf_xinetd" > /etc/banner_fail
RUN chmod +x /start.sh
COPY ./bin/ /home/ctf/
RUN chown -R root:ctf /home/ctf && \
    chmod -R 750 /home/ctf
CMD ["/start.sh"]
EXPOSE 9999

start.sh

It starts the xinetd service.

#!/bin/sh
/etc/init.d/xinetd start;
sleep infinity;

ctf.xinetd

When you set up a netcat server using nc -lvp 8000, it sets up a listener on port 8000. However, only 1 user can connect to this netcat server at a time. Therefore we use xinetd, which allows multiple netcat connections simultaneously, and kills the processes once the connection is closed.

To configure xinetd, we need a file called ctf.xinetd containing:

service ctf
{
    disable = no
    socket_type = stream
    protocol    = tcp
    wait        = no
    user        = root
    type        = UNLISTED
    port        = 9999
    bind        = 0.0.0.0
    server          = /usr/bin/python3
    server_args     = /home/ctf/encrypt.py
    banner_fail = /etc/banner_fails
    # safety options    
    per_source  = 10 # the maximum instances of this service per source IP address    
    rlimit_cpu  = 1 # the maximum number of CPU seconds that the service may use    
    #rlimit_as  = 1024M # the Address Space resource limit for the service
}

We set the sockets to stream and follow TCP. For challenges using other interpreters to run, change the server and its args.

Build the container

To build your container, execute the following command:

docker build -t <challenge-name> <path-to-challenge-directory>

Note: use sudo before docker, if your user is not in the docker group.

Now, once you built the container, you have to run the container. You can do this using docker run.

docker run -d -p <external-port>:<container-port> <challenge-name>:latest

Here, the <external-port> represents the port on your computer (or the port to be exposed by the server) and the <container-port> represents the port exposed from inside the docker container. Using -d option for it to run in the background. Test the chall by running it on the server and visiting it with netcat : nc <server-ip> <port>.

To display all the running containers, docker ps -a

To stop and remove your container respectively, you can run the following command:

docker stop <container name>
docker rm <container name>

With this, I end the blog. Thanks a lot for reading and I hope it helped you nicely.

Follow shellpwn for more interesting blogs on Cybersecurity and CTFs:

shellpwn