Part 3. Persisting data in key-value store using GenServer — Elixir/OTP

Photo by Kolar.io on Unsplash

This is part 3 of the 7 part series. Please read the previous articles to catch up on the topic.

So the problem with the previous implementation is that whenever you restart your server the data stored in the server state is lost because we have not implemented a way to persist the values in a file.

In this article, we are going to learn how are we gonna incorporate a strategy to persist the store values and fetch them when we restart the server. We are going to persist the values in a file that will act as our makeshift database but this will do for now as to just understand the logic.

Let’s get started 🏎️

Before beginning make sure you have Elixir and Mix installed in your system.

Now create a mix project by typing

mix new ex3_key_val_with_db

This command should create your project structure like this

Mix project folder structure

Delete all the files in the lib folder and create a file with name database.ex. Also, copy the other files from the 2nd exercise into the lib folder i.e.manager.ex, server.ex and store.ex.

Now, we need to implement a strategy where whenever a store is created the server should check if a file with the same store name exists on the local storage present. If yes then it should copy the contents and load it in the internal state of our server. Also whenever an update is made to the state of the server, then the state should be persisted in the database as a file so that upon restarting the server we get back the last state stored in the server.

The image shown below adds a new entity called database which is again a GenServer that deals with storing and updating files as the store gets updated.

Design of system

Show me some code 👨‍💻

Here we now have 3 entities to care about.

1. Database
2. Manager
3. Server
4. Store

database.ex

The database process is a GenServer which is responsible for creating a folder named “database” if not present when it is started.

Whenever the process is started it will check if the folder is present or not if not present then it will be created.

Furthermore, it has two main functions. The first is to fetch a file from the database folder which would be named after the store created, read the content of the file and convert to the contents of the file to a Map data structure. The second function of it is to convert a Map data structure to binary form and store it into a file inside the database folder.

manager.ex

The function of the manager is same as before to keep track of the ongoing server process with their names and PIDs in its internal state and to create stores when requested.

server.ex

Now in the server process, there are 2 main changes that we can see. Now, when the server is started, in the init block we consult the DB module to check if a file with the same store name is present on the local storage or not. If yes then the content of the file is converted to a Map and is loaded in that server’s internal state. This is how if data is present it gets loaded to the store.

The other function is to issue a command to the database module to write the server’s state to a file whenever a put or a del command is issued to the server process.

store.ex

The store file is same as before.

The final run 📟

In order to start the whole system, we first initiate an elixir session
by typing

iex -S mix

session 1

As you can see in session one we first start our manager process, followed by the db process. Then we create a store with the name store1 and put a value inside it. As soon as we do that a file is created in the database folder with the same name as that of store.

new file with store name created

Now, we exit the iex session and start another session by typing in the same command.

session 2

Now what we do is that we do the whole process until creating the store with the same name. But this time when we try to get the value for a key which was stored in the previous session we get back the same saved data. This means that the data was successfully written to the file and was retrieved when we instantiated a new server process with the same store name.

So this is how we successfully added persistence in the key-value store.

I hope this post has helped you get a little bit better understanding of the whole process.

The complete source code of all the parts are here.

References 📝

1. Elixir in Action. 2nd Edition.
2. GenServer behaviour docs.