Reading docker container logs with Golang docker-engine API

We released a new feature on Geektrust yesterday, where developers can submit solution to one of our coding problems online. There is an IDE like editor and they can write code on our site. To execute this code, we create an instance of a docker container each time.

The idea was to execute the code in the isolated container and log the output/error in JSON format, and then read this log to send the response back to the developer who is executing his/her code. A REST service will receive the developers code and language in which he/she has written. This service is written in Go and I used Docker’s docker-engine API for creating & running the container. We read the logs using the same API.

However there is a catch while reading the logs from stdout stream of container. The data written by the container is in a format which has a HEADER & PAYLOAD. The header contains the information about the stream and payload and is encoded on the first 8 bytes like this.

header := [8]byte{STREAM_TYPE, 0, 0, 0, SIZE1, SIZE2, SIZE3, SIZE4}

If you read the container logs like this

then you will see unicode characters printed on the console and then the actual payload. If you expect a JSON from the console and is ready to decode the JSON to a struct like this

json.NewDecoder(strings.NewReader(string(content))).Decode(&myStruct);

then this will fail, because it is reading data which contains unicode characters and JSON.

So we have to skip the first eight bytes of the stream and read from the reader like this.

When I wrote the code first (like in the first gist), I was not aware of this issue and I couldn’t get what these unicode characters were. The documentation on the remote api for getting the container logs doesn’t have a mention about it. It was this issue raised on Github that pointed me to docker remote-api’s Attach Container documentation which explains about HEADER & PAYLOAD. Hopefully some body will update the remote API’s documentation.