0-day bug breaks multi-million dollar system

Blazej Adamczyk
Dec 30, 2020 · 11 min read
Image for post
Image for post

The other day we got an order for black-box pentests from a country-wide organization. The organization serves tens or even hundreds of services across the country and their users are counted in millions. To be honest, looking from the impact perspective this is one the biggest hacks we’ve done so far. But this is not the main reason I decided to publish this write-up..

The real reason is that the hack itself is, in my opinion, really interesting! This is the type of hack you can be proud of. It took an incredible amount of time and many different approaches before it succeeded. Thanks to my stubbornness I didn’t give up too fast and finally, I’ve found four new vulnerabilities which together lead to unauthenticated, remote, and arbitrary code execution inside of the organization.

Coming to the details. Briefly speaking the target was a set of purpose-built web applications used to fill, process, and send many different types of forms. As the tests were black-box we didn’t get any information about the system, nothing more than the end-user is able to see. This is important because finding and fully exploiting bugs in a purpose-built backend obviously isn’t too easy.

As usual with pentests we started with listing all the servers and endpoints looking through the process flow and front-end code. In black-box tests, it is a really important stage as you may learn about some libraries or components the system is using which are typical off-the-shelf products. This can tell you more about the architecture and technology but also may allow you to identify if the system contains publicly known vulnerable component versions. Looking through the gathered lists unfortunately we haven’t found any vulnerable product versions but one component — especially the way it was used — was really standing out among the others.

This suspicious component was accepting html file from the browser (user input) and was converting the html file into PDF. Such services seem very interesting from the pentester’s point of view as (1) there is some file upload; (2) some processing (parsers etc.) and we also (3) retrieve some results back. All the steps can be vulnerable and in fact, this was the case.

One more information — the component was exposed by its name in the url path. The path finished with: https://……../api/gotenberg. I myself wasn’t aware of that component but the name was so distinguishable that a quick search for “gotenberg pdf” resulted in a github page with an open-source PDF conversion component — Gotenberg, which by authors is advertised as:

A Docker-powered stateless API for converting HTML, Markdown and Office documents to PDF.

Side note: The vulnerabilities described in this article (CVE-2020–13449, CVE-2020–13450, CVE-2020–13451 and CVE-2020–1345) were reported to authors and were very quickly fixed. I want to thank Julian Neuhart for fast reaction and fix. The authors agreed and stated that Gotenberg in this major release (6) should only be used internally by trusted applications not exposed to the external world.

As it resulted indeed the whole system was built using microservices architecture and among many others, there was this Gotenberg component fully exposed to the outer world.

In fact, the system was using only the HTML conversion API but the way it was exposed allowed all unauthenticated Internet users to use the whole API — HTML, markdown, and office, including the /url/ endpoint which allowed to point the component to an external web url over the Internet to generate the PDF file. This by itself is a vulnerability because allows to anonymously proxy web traffic or even use it to perform DoS attack using our client’s infrastructure.

Because the component was open-source and available in Docker hub it was quite easy to test it for very basic attacks. Thus after starting the Docker container using (I’m purposely starting here vulnerable version 6.2.0, as this was already fixed in 6.3.0):

$ docker run --rm -p 3000:3000 --name gotenberg thecodingmachine/gotenberg:6.2.0

We can play with it and try to find some interesting attacks.

The first two vulnerabilities were trivial and were found very quickly. But the way of exploiting them was much more challenging.

First vulnerability — upload arbitrary files

If the component accepts files (upload) then one of the first tests a pentester does is trying the path traversal attack. Thus looking at the documentation we can call the /markdown/ endpoint like this (note the bolded part):

$ curl 'http://$URL_GOTENBERG/convert/markdown' --form files=@index.html --form "files=@test;filename=../../../tmp/test" -o res.pdf --header 'Content-Type: multipart/form-data'

Surprisingly — because in most web frameworks this attack will not work as the filename parameter input is being escaped at the framework level — Gotenberg was actually vulnerable because it is written in Go lang and is using net/http package to handle http requests which is not escaping the filenames before passing it to the application, the latter was not escaping it either. Thus the above curl writes the file test in the path /tmp/test in the Docker container. This can be verified by running:

$ docker exec -it gotenberg bash -c "ls -la /tmp/"
total 16
drwxrwxrwt 1 root root 4096 Dec 29 19:10 .
drwxr-xr-x 1 root root 4096 Dec 29 19:10 ..
drwxr-xr-x 2 root root 4096 May 1 2020 hsperfdata_root
-rw------- 1 gotenberg gotenberg 60 Dec 29 19:10 test

Second vulnerability — arbitrary file download

Just a little more tricky was the second vulnerability. According to its documentation Gotenberg allows to convert markdown files — but the way this is done is through an HTML file that renders a template. The templates can call a helper function toHTML which actually reads any file in the file system and renders the markdown to html.

This, as you can probably guess, can lead to arbitrary file disclosure within the container. By itself, this is not a very critical vulnerability as there are not many things you can get with it, but as you will see later this vulnerability can be used to do more damage!

So, to see how it works let's create an HTML (let's name it index.html) file:

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>My PDF</title>
</head>
<body>
<pre style="white-space: pre-wrap;">
Path:
{{ .DirPath }}
PASSWD:
{{ toHTML .DirPath "../../../../etc/passwd" }}
IP:
{{ toHTML .DirPath "../../../../proc/net/fib_trie" }}
TCP:
{{ toHTML .DirPath "../../../../proc/net/tcp" }}
env:
{{ toHTML .DirPath "../../../../proc/self/environ" }}
</pre>
</body>
</html>

And call Gotenberg like this:

$ curl 'http://$URL_GOTENBERG/convert/markdown' --form files=@index.html -o result.pdf --header 'Content-Type: multipart/form-data'

In the result.pdf you get a nice pdf file containing some nice information like the container’s passwd file, IP addresses, TCP connections and environment variables. This might be quite interesting but is not very critical (yet!).

The fun part — code exec

Okay, we’ve got the upload and download vulnerabilities but we want more.. First, let's talk a little bit about Gotenberg internals. It is a Go lang http server that runs within a Docker container that uses different off-the-shelf components to convert different formats to PDF.

For converting HTML, markdown, and external URLs it uses the Google Chrome browser running in headless mode. This engine is being controlled using (the same as it is done in some automated testing frameworks) Chrome DevTools Protocol by the Gotenberg process which is redirecting the headless Chrome browser to the appropriate files/URLs and finally printing the rendered HTML to PDF.

Important note: Having the possibility to redirect built-in Chrome to any URL the attacker is from the beginning having slight access to the localhost of the container (and this port 9222 where the DevTools listen) but also to other machines in the internal network (docker containers or internal IPs of the client system). This proves that Gotenberg is rather an internal interface that should be hidden inside of the system and if its API should be exposed to the outside world it should be stripped to very limited functions.

For office documents, it is using the unoconv Linux package which underneath runs libreoffice, renders the office document, and as previously prints it to PDF.

Having the file upload vulnerability the first thing which came to my mind is whether there is a file in the container which can be overwritten which would lead to code execution.

After jumping into the container and looking through the file systems for writeable files:

$ docker exec -it gotenberg find / -writable
/dev/core
/dev/stderr
/dev/stdout
/dev/stdin
..
/gotenberg/.bash_history
/gotenberg/tmp
/tini

I see a file in root directory which is writable called /tini, after having closer look it is an executable file, and it is the one primarily called by Docker container at start (so called entrypoint — see it in sources). The tini is a tiny program that can be used in a Docker container properly handling the real running process by taking care of signals, zombie processes and so on. As it occurs in Gotenberg this file is writable by user “gotenberg”. This seems like a viable way to the final code exec.

Idea: we can use the file upload vulnerability to overwrite /tini executable and the next time the container is going to be started it is going to run our executable.

Even then there is a quite important catch: Gotenberg authors propose to run the container in a “temporary run” (notice the --rm switch in the above docker run command). If the container is started this way it means it will never get stopped and restarted again so unfortunately, this attack would have a very limited application.

After trying the above idea it occurred that Gotenberg while uploading a file is changing its permission to 644 which makes the /tini executable lacking +x permissions and thus even if the container would get stopped and restarted it would not run because of lack of execute permission.

Nice try but no luck..

While listing the writable files I’ve found out that there are two user profile files that can be also overwritten and seem pretty interesting, these were:

/gotenberg/.local/share/applications
/gotenberg/.local/share/applications/mimeapps.list

Remember that Gotenberg runs Google Chrome? If you are using Linux you probably know that the above applications folder and mimeapps.list file are responsible for default applications for handling different content.

Idea: what if we create an application x.desktop file in the applications folder which would run our payload, e.g.:

[Desktop Entry]
TryExec=bash
Exec=bash -c "echo asd>/tmp/hacked"

And then we hook this application to handle a custom URL scheme using the mimeapps.list:

..
x-scheme-handler/mailto=x.desktop
..

The above line in mimeapps.list registers our x.desktop application with the mailto: URL scheme, thus each time the browser is going to call mailto: link it will start our x.desktop application.

Trying this idea out on my non-headless chrome works well, going to the mailto:test link would actually execute the appropriate x.desktop. It works even through the remote DevTools Protocol debugging.

Unfortunately, this attack does not work when running in Gotenberg. It seems this is probably because of the way chrome is being initialized in headless mode where the custom URI schemes (using Linux xdg) are not being properly called called. Indeed visiting the mailto: scheme on my local machine with headless parameter does not call the proper application either.

Again, no luck..

Being a step from giving up I decided to try one last time. I started looking at the last piece of the puzzle — i.e. unoconv and LibreOffice. First I looked through the way unoconv is being called and if there is any way to inject code etc. Unfortunately, I couldn’t find anything useful there.

Then I thought about the old good macros. Maybe we can somehow inject a macro to the file and magically make it run. Unfortunately, all the quick tries failed (as expected) because LibreOffice does not allow to run macros until they are running from a trusted location which /tmp/ is of course not.

But wait.. where is the trusted location saved by LibreOffice? Is it somewhere in the user home directory? I’ve quickly checked on my local machine and indeed the trusted macro paths are added to the ~/.config/libreoffice/4/user/registrymodifications.xcu file as line similar to:

<item oor:path="/org.openoffice.Office.Common/Security/Scripting"><prop oor:name="SecureURL" oor:op="fuse"><value><it>PATH</it></value></prop></item>

Hmm, looking in the gotenberg home I don’t see the .config directory.. But then I reminded myself that while looking through the parameters how unoconv is being called by Gotenberg I saw something about user directory.

After looking one more time I realized that Gotenberg runs each conversion using a fresh temporary user directory (in /tmp/ path; with --user-directory parameter). Really interesting..

What is more, after running Gotenberg for a while and trying different office conversions I realized that these temporary folders have pretty short names and they are not removed after the request!

Looking at the temporary folder names all were in between standard TCP ephemeral port range (~30000 up to ~65000) which seemed very predictable.

Indeed looking at the code we see that Gotenberg calls freeport golang package and this is further calling net package which creates a new socket and binds it using bind syscall without specifying the port number — this according to Linux documentation is causing Linux kernel to pick a random port from the ephemeral port range defined in /proc/sys/net/ipv4/ip_local_port_range. This is the ephemeral port randomly selected by the kernel while creating a new socket. Analyzing kernel code we can see that the function inet_csk_find_open_port is responsible for choosing a port from the ephemeral range. The default Linux kernel range is 32768 — 60999 and the syscall bind takes only the odd numbers. Thus we have only about 14115 possible port numbers.

So, we know that the folder names are pretty predictable and what is most important are not removed at the end.

Idea: The idea is to first create a profile without running macros (we don’t know what the profile number is so we have to “flood” and check until we find a profile with the number we want.. this can take some time). Then upload a new LibreOffice configuration file which somehow runs the macro while a file is loaded with this profile. At the end “flood” the server again with requests until it reuses the same port number one more time causing reusage of the prepared profile.

Instead of using a trusted path I’ve found that we can pretty easily set up global macros which can be run after LibreOffice opens a document. This is even a better approach as it does not depend on the file which was uploaded. E.g. we can leave normal users to “flood” the server for us.

Indeed this idea worked! The exploit executes quite long before but it pretty stable.

What’s next?

Finally, having code exec we could start to pivot attacks to clients' internal infrastructure which allowed us to connect to internal important services like orchestrators and other crucial management services (e.g. config server) as well as database engines and other microservices.

Internally client had no firewalls between services and additionally the monitoring did not notice our reverse meterpreter session for about two weeks until we closed it on our side!

This leads us to a conclusion:

  1. Do not expose too many endpoints to the outside world.
  2. Separate services according to their needs (e.g. with firewalls).
  3. Allow only approved inbound and outbound connections to your infrastructure.
  4. Monitor the traffic for some unusual and long-lasting connections.

Video of running exploit:

Original vulnerability report and disclosure:
https://sploit.tech/2020/12/29/Gotenberg.html

Exploit code:
https://github.com/br0xpl/gotenberg_hack

InfoSec Write-ups

A collection of write-ups from the best hackers in the…

By InfoSec Write-ups

Newsletter from Infosec Writeups Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

Blazej Adamczyk

Written by

Security researcher focused on software and networking. Well oriented in operating systems, web applications, networking, cryptography and virtualization.

InfoSec Write-ups

A collection of write-ups from the best hackers in the world on topics ranging from bug bounties and CTFs to vulnhub machines, hardware challenges and real life encounters. In a nutshell, we are the largest InfoSec publication on Medium. Maintained by Hackrew

Blazej Adamczyk

Written by

Security researcher focused on software and networking. Well oriented in operating systems, web applications, networking, cryptography and virtualization.

InfoSec Write-ups

A collection of write-ups from the best hackers in the world on topics ranging from bug bounties and CTFs to vulnhub machines, hardware challenges and real life encounters. In a nutshell, we are the largest InfoSec publication on Medium. Maintained by Hackrew

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store