Ruby software development involves building and shipping software in packages or libraries. In the Ruby world, these packages are called gems and they are usually published on RubyGems. Developers from around the world can search for these gems, install them, and integrate them into their own projects. RubyGems also provides a nice command line to manage the gems, allowing users to push and yank gems.
So far, this is nice, but what if there is a need to develop some private gems that should not be available for everyone in the world? Unfortunately, this is not supported by RubyGems and there is a need to host a private gem server to be able to share the private gems internally among the development teams.
In this piece, I will present my attempt to host my own Ruby gem server. I will cover the following points:
- What are the alternatives to RubyGems?
- A description of my solutions for hosting a Ruby gem server.
- Further improvement for the solution.
Looking for an Existing Solution
Before I start developing any code or solutions, I usually take some time to search for alternatives or existing solutions that I can reuse to achieve my tasks. We do not always need to reinvent everything from scratch. After some quick research, I found this piece that describes a couple of options for hosting your own
gem server. These options are presented below with a short description of each of them.
- built-in gem server: This server comes with
Rubygemsand can be only used for viewing the installed gems. It does not support pushing or yanking gems.
- Gemirro: This gem server focuses on mirroring and also does not support pushing or yanking gems. It is a good choice if you would like to configure your pipelines and applications to a private gem server that takes care of downloading and cashing gems from RubyGems. So if RubyGems is down, you will still be able to install gems from the internal server.
- Gem in a Box: This is another existing open-source gem server. Fortunately, this server supports both pushing and yanking gems from the command line as well as uploading and deleting gems from its web UI.
It was clear for me at this point that the best choice for achieving my task would be Gem in a Box due to the fact that it supports more features than the other options. Therefore, I started digging deep into the project to understand how I could start hosting my own Ruby gem server with Gem in a Box and what features were supported by the projects. As a result, I came up with the following list of built-in features and a workaround for some issues:
- The project provides a web UI that allows viewing, searching, uploading, and deleting gems.
- The project provides a command-line tool called
- The command
gemcan be used to push and yank gems from the server.
- The wiki for the project describes a workaround to extend the project to support
HTTP BASIC AUTH,
Alternative UI styling,
ssl configs, and other features.
Unfortunately, these features were not enough for my case and I was missing the following features:
- LDAP role-based authenticating and authorization.
- API key generation.
- Support for the
- Support for the
Therefore, I decided to extend the Gem in a Box project with the missing features with the help of some solutions posted in the project’s wiki.
Implementing the aforementioned features means that there is a need for persisting some data. We at least need to store the API keys for the users and the owner of each of the gems. Therefore, I started exploring my options here, and since I want to make it as simple as possible and without any integration of any databases like MySQL, MongoDB, etc. I finally decided to persist the data in text files.
Writing software to manage data in a text file was beyond my scope. Therefore, I took some time to find an existing solution for this feature. Luckily, I found that Ruby provides two classes to do this task, and they have a nice and easy interface. These classes are:
PStore: Implements a file-based persistence mechanism based on a Hash.
YAML::Store: Provides the same functionality as
PStore, except it uses YAML to dump objects instead of Marshal.
Because both of the classes use the same interface, I decided to support both of them in my solution and allow the option to choose between them via an environment variable.
For simplicity reasons only, I decided to make the simplest data models that I need to implement the intended features. I came up with the following threes models:
- User: This model persists the following user data (
- ApikeyIndex: This model correlates the API keys with the users. You may say that the API key is already correlated to the user in the User model. Yes, that is true, but since we are working with Hash objects here, it would be easier to have a reverse index instead of looping over the user’s objects to find the user API key.
- GemIndex: This model correlates the pushed gems with their owners (the first user who pushed the gem).
My requirement for role-based authentication and authorization is very simple and can be covered by the following points:
- Downloading and viewing gems is open to everyone without any authentication.
- Pushing, uploading, yanking, and deleting gems is protected by LDAP authentications.
- Two LDAP groups are supported for authorizing actions (maintainer and admin).
- Maintainers can push/yank their own gems only.
- Admins are allowed to push/yank any gem.
Since Gem in a Box uses Sinatra, it also uses Rack and Rack middleware. Therefore, I decided to extend the solution by implementing middleware classes that handle each of the needed requests. Below is a description of the implemented middleware classes and their usage.
This middleware provides a static health check endpoint that is available on the root route of the application: http://localhost:8080/health. This endpoint returns a static string of
OK. It can be used to check if the application is up and respond to
Unfortunately, the endpoint
signup is not implemented in Gem in a Box. Therefore, I wrote this middleware to handle the requests and collect user data. This middleware performs the following actions:
- Ask the user to enter their LDAP credentials. This is supported by
- If the user credentials are valid, the middleware will create an
api_keyfor the user (only if they do not have one yet), persist the user info (LDAP groups, username, API key) in the user store, and persist the
api_keycorrelation in the API key store. Finally, it will show the API key for the user.
- If the user credentials are not valid, it will keep asking for entering valid credentials.
Signing up multiple times will not generate multiple API keys for the same user. The middleware will generate a user API key only if the user has no API key in the store. On the other hand, LDAP groups will be updated each time the user signs up.
This middleware does almost the same thing as the
SignUp middleware, except it handles the API request
/api/v1/api_key and not the signup one. This endpoint is requested whenever a user tries to sign in using the
gem command line or when a user is trying to push a gem without signing in first.
gem signin --host http://localhost:9292
gem push myge-0.11.2.gem --host http://localhost:9292
As a result of successful LDAP authentication, the middleware will persist the user and
api_key data to the related store objects, and it will return the
api_key in a text format so it can be added to the local credentials file.
➜ : cat ~/.gem/credentials
Gem in a Box provides two API endpoints for pushing and yanking gems from the server. These endpoints are:
POST /api/v1/gems: For pushing gems to the server. This endpoint will be requested whenever the following command line is used.
$> gem push mygem-0.11.2.gem — host http://localhost:8080
DELETE /api/v1/gems/yank: For yanking (deleting) gems from the server. This endpoint will be requested whenever the following command line is used.
gem yank mygem -v 0.11.2 --host http://localhost:8080
ApiGem middleware intercepts the API calls for the requests above and performs the following actions:
- Validate the presence of the API key.
- Validate if the API key is allowed to push or yank the gem.
GemIndexto correlate users and gems. If the performed request is the
POST /api/v1/gemsrequest and the gem is a new gem, then the middleware will add an entry on the
GemIndexstore. On the other hand, if the request is a
DELETE /api/v1/gems/yankrequest and this is the last version of the gem on the server, then the middleware will remove the index entry from the
WebRequestsLdapAuth middleware intercepts three web endpoints provided by Gem in a Box. These endpoints are listed below:
GET /upload: This request is used to view the upload gem from the UI. The middleware only protects this endpoint with LDAP authentication and does not perform any other actions.
POST /upload: This endpoint is requested when the user hits the upload button on the upload web form.
POST /gems/.*gem: This endpoint is requested when the user hits the gem delete link from the main page.
For the last two requests, the middleware will perform the following actions:
- Ask the user to enter their LDAP credentials.
- Verify if the LDAP user is allowed to manage the corresponding gem.
- Update the
GemIndexstore so it is the same as with the
To finalize the solutions, I wanted to add support for Docker. Therefore, I added to the solutions a Dockerfile to be able to generate Docker images for the gem server. These images can be used to start the gem server in a Docker container. Below is an example of a
docker-compose file for the gem server with all the supported environment variables.
If you would like to try the gem server, just copy the above
docker-compose template, modify the LDAP configs, and execute the following command:
$> docker-compose up -d
Running a gem server is a must if you would like to start developing private gems. There are a couple of solutions for running your own gem server. Currently, Gem in a Box is the most advanced one and the closest to RubyGems. However, the project is still missing some security features like protection against malicious code injections (anyone who has access to the server can push, upload, or delete gems ) and role-based authorization.
This piece proposes a solution for extending Gem in a Box and supporting the mentioned features. The complete source code solution can be found on GitHub and the Docker images for the gem server are published in this repository. The solution still needs some improvements and enhancements, and any contributions are more than welcome.