EMBED OTHER BINARIES IN GOLANG BINARY

-Migrated from my original blog

BACKGROUND

This just started as a thought… What if I could compile an existing executable binary into a golang binary. So in my mind I toyed with the idea, and so far the best route I could find (after 10min of thinking), was convert the binary to a byte array. Now this seems very inefficient, and it is, but still fun and kind of cool.

WARNING

With this tutorial we will be using a simple small binary, but when I first thought of the idea I tried it with a NodeJS binary. The NodeJS binary is 28.8 MiB (ver 6.7.0). When I converted the binary to a byte array the generated source file was 126 MiB. To top that I ran it on my laptop, running 12 GiB of ram and 16 GiB of swap on an SSD. I use so much swap because I may run a couple VMs at a time, and I am generous of memory I give them.

Now with the NodeJS binary the go build command I ran took forever, and was eventually killed because I ran out of memory. Yes that included filling the swap.

So if you want to try this, just note it may take up a bit of memory. With the code, and binary we will be using it shouldn’t be to rough, but be warned! Maybe down the road I or someone else will figure out a more efficient method.

Also note I am on Linux and have not tested on other OS’s, so this may not work on OSX, Windows, ect…

FILE STRUCTURE

./
├── generators/
│   └── generator.go
├── main.go
└── sample-data/
    └── main.go

2 directories, 3 files

THE SAMPLE BINARY

With this tutorial we will just create a simple hello world. The code really needs no explanation, and if it does to you, then the rest of the tutorial may be to advanced for you and you should start with a Tour of Go

Then we run:

$ go build -o ./sample-data/{generated-bin,main.go}

This gives us an executable binary named generated-bin. This will give us the binary that we will compile into another go binary. This is just an example and you could use any language to create an executable, just make sure it’s not too big.

CREATING THE GENERATOR

The generator will be more of a “script” written in go. We actually won’t create an executable from this, but call it from our main.go source file using the go generate command.

Nothing too complicated, just takes the executable and converts it to a byte array and then creates a go source file in the root directory, because we are calling it via the go generate command so it executes this file as if its working directory was the root directory. This will create a source file in the root directory called generateddata.go. This has a package main so it will be compiled with the main build.

THE MAIN APPLICATION

This is a VERY simple bit of code.

Very Important: make sure to include the go:generate comment, or this wont work

The go:generate comment tells the go generate tool that it need to execute the ./generators/generate.go “script”.

BUILDING

Ok assuming that you built the generated-bin executable, go ahead and run the next few commands.

First we run the generate command:

$ go generate

If you run ls you will see the generateddata.go in the root directory. Now here is another word of warning, don’t open this file with most text editors like Atom, and even vim had a hard time (part of the reason is because I use YouCompleteMe). If you want to inspect the generated file use a very minimal command line tool like less. The byte array is pretty large.

Now let’s build it:

$ go build -o main *.go

Now depending on the size of the generated file this command will take some time, and be CPU and memory intensive. With our example binary it shouldn’t be too bad.

Now with the executable built, we can run it:

$ ./main

This creates a new executable in the root directory called final-bin. When you run ./final-bin you will see that it outputs the same as the executable created in the sample-data directory.

An idea like this could be used to embed things like electron binaries, images or anything. While this is a “cool” idea, it is impractical, until we find a better way to statically save the data.

Please feel free to comment on how to improve on this idea.

Link to the github repo: embed-binary