Journey of leaving comfort Ruby world to understand and create own native gems — introduction

Almost every Ruby programmer have seen this message during gem install or similar message Installing xyz 1.2.3 with native extensions during bundle install. For some programmers this message can increase heart-rate or induce a feeling of panic usually followed by praying to our god (Matz). Thousands of hours were already spent on googling following error messages, looking up for missing platform-specific dependencies, installing “dev packages”, looking up for right combination of install parameters or asking television oracle on the phone to finally find the solution and get one of those satisfying messages -> Successfully installed xyz-1.2.3 or Bundle complete!.

Those problems are usually first introduction to native extensions in Ruby and (since it is not nice and friendly introduction) users often tend to not be comfortable working with native extensions. In this series I would like to introduce ecosystem around Ruby native extensions giving warm introduction to anyone interested in this mysterious part of Ruby ecosystem.

Why?

With Ruby native extension you can use any library from huge amount of great (and usually heavily optimized) (not only) C libraries even if there’s no Ruby binding written already! Understanding native extensions under the hood will make it easier for you not only to solve unexpected problems but it will also open you doors for bunch of new possibilities of solving coding challenges.

Disclaimer & requirements

There’s no Ruby code in this introduction chapter. But don’t worry, there will be a lot of Ruby code in following chapters. All code will be present at public GitHub repository.

There is some C code in this introduction chapter. I’m not going to explain C in details. But don’t be scared if you’re not C-friendly programmer. We will only declare variables and use functions. No scary pointers involved!

Please take in mind I’m not experienced C programmer, I’m just curious Ruby programmer. I would like to share virtual journey you can experience getting out of your Ruby comfort zone for a while. I would like to show you a little different approach on fictional, but still nearly common and usual task we need to solve daily as a Ruby programmers — using 3rd party code libraries.

I will use trivial example of writing Ruby binding to C library generating unique identifiers (UUIDv4). Overall approach taken is used only to describe and explain Ruby native extensions. There are much better ways how to generate UUIDv4 in Ruby. TL;DR SecureRandom.uuid for example.

You will need gcc and libuuidinstalled locally. I’ll try to help you later in the text installing those. Anyway code in this first chapter will probably not work on Windows. No worries, we will get back to Windows support later in one of the following chapters. It is still worth it to follow this chapter if you’re on Windows to get the brief introduction.

Let’s start the hard way — leaving comfort Ruby world

Have you ever heard about UUIDs? Do you know there are 5 versions of UUID? In this series I’ll try (with you) to write a gem generating UUIDv4 at good speed. UUID is described in RFC 4122, let’s start with reading it and start to think how to implement it in performant way…

Wait!

Isn’t there any library already doing this? Hmm. C libraries are usually fast. C libraries usually starts with ‘lib’. Let’s try to find out if there’s not any libuuid library already done.

Initial Google search takes us to https://linux.die.net/man/3/libuuid. Quick click on uuid_generate link. And finally I seeuuid_generate_random function which is probably what we are looking for - UUIDv4 (random) generator.

NOTICE: I was able to extract a lot of useful info from single on-line “man page”. Take a look what’s man page, how they work and how to access them if you’re not familiar with them already.

Initial proof of concept

Let’s take this opportunity to leave a nice safe Ruby world for a while and start a new interesting experience in C world!

Sometimes I like to get some initial PoC done quickly as possible. Instead of reading all readme pages carefully I’m just trying to find out some example code to play with. Using some basic google-fu takes me to libuuid example gist. After few tries I’m able to extract basic usage of uuid_generate_random function into uuidgen.c file even having really low C knowledge.

NOTICE: If you don’t understand following C code with my inline comments, it will be good idea to take a quick break and try to self-study some C basics. Successfully finishing first two short courses at https://www.learn-c.org/ should be enough.

uuidgen.c test file

To successfully execute this code once it is compiled we need to link it with libuuid library. Let’s use gcc compiler for that.

gcc uuidgen.c -o uuidgen -l uuid

• gccis compiler command
• -l uuid tells compiler to link our compiled file to libuuid library (we don’t need to specify ‘lib’ part)
• -o uuidgen specifies output file

Sounds easy, ugh?

Let’s compile some C, YAY!

If you’re lucky enough, you have successfully compiled example C code.

Yes, finally we can say that! (https://xkcd.com/303/)

But since we have not installed any required dependencies to compile our code, command specified above will probably fail and you will need to fix few problems first.

gcc not found

If you see this, it’s sad to say, but you’re missing C compiler.

• on Linux it will be packaged in your distribution repo
  * on Ubuntu -> apt-get install gcc
  * on Fedora -> dnf install gcc
• on Mac you can use Homebrew -> brew install gcc.
• on Windows gcc can be installed with Ruby itself as a part of devkit

no worries, we can fix it

fatal error: uuid/uuid.h file not found / cannot find -luuid

Once compiler is in place, you will probably miss libuuidlibrary itself and you’ll get error. To solve this we need to install libuuid library and it’s C headers to your system.

• on Linux install libuuid development package
  * on Ubuntu -> apt-get install uuid-dev
* on Fedora -> dnf install libuuid-devel
• on Mac you can use Homebrew brew install ossp-uuid.

You can think about this step as a analogy to gem install --development xyz in Ruby world.

TIP: If you are not familiar with any part of that gem command feel free to check rubygems dependencies and gem install --help. We will take a deeper look at that later as well.

If you wonder what to do on Windows here, it is probably possible to install or build libuuid as well. I would skip Windows compatibility here to get back to it later in one of following chapters.

compiling for real

Now we should have both dependencies (gcc and libuuid) installed. Let’s try to compile our C example code again.

gcc uuidgen.c -o uuidgen -luuid

Once code is successfully compiled, running ./uuid-gen should print random UUID to standard output of your console now.

If you’ll have any troubles compiling this code, please let me know in comments or on Twitter and I’ll try to find out how to help you.

PROTIP: Do you remember that return statement at the end of our C code? It is called exit code and it is usually stored in $?environment variable. It is good to remember this since we will reuse it later during CI configuration.

Yes! Finally we have some result!

Isn’t it exciting to get out of Ruby comfortable world for a while and take a sneak-peek into dangerous C code world?

We did a great job. Now we know there’s libuuid C library doing exactly what we were looking for. We know how to use it in C world and we also know how to install it on our system. All that covered by PoC C code generating our desired UUIDv4 and printing it into console. There's huge chance this library is well performant since it is written in C. But we have no evidence for this for now. No worries, we will take a look at performance benchmarking soon.

Slowly getting back to Ruby world

With our new knowledge how to use libuuidlibrary we can slowly start to think how to use it in Ruby. Once that will be done, we can go back to our comfort Ruby world where we have great tools we understand (including great benchmark-ips gem) to take a look how we do compared with other possible ways of generating UUIDv4 in Ruby.

In next chapter we will finally move back to Ruby world and bootstrap basic rubygem skeleton with native extension support. We will also write initial README.md as part of RDD technique, write initial automated test suite as part of TDD technique using Minitest and use Travis CI to run our test suite on Linux and Mac platforms.

When this journey will continue?

If you would like to follow this journey please let me know in comments or on Twitter. I’m not great writer and also I’m not English native speaker. Showing some love and kind response will be great motivation for me to continue writing down this journey.

In following chapters I would like to cover following topics:

• built gem compatible using TDD and RDD with UNIX (Linux, Mac, …) platforms tested on CI (Travis CI)
• benchmark native extension and compare to other solutions
• introduce Windows support tested on CI (Appveyor) and explain some common tricks (fat binary gem releases, …)
• explain how to spot and detect memory leak in native C extension using Valgrind memcheck tool
• introduce JAVA (JRuby) platform support tested on CI (Travis CI) as well
• show alternative ways how to use the same C library in Ruby world (FFI, SWIG, …)
• build the same native extension in another languages (PHP, Lua, Node.js, Python) for fun and profit
• any crazy idea based on your feedback here

If you consider this article useful, please share it. If you have any suggestions or problems running any code, feel free to ping me in comments or directly on Twitter as well. I’ll try to help you if I will be able to. My plan is to update this article with feedback I’ll gather to make it easier to follow for future adventurers.

I hope we will meet soon again to continue this journey!