How to make Redis play nice with your data

I love using Redis in my projects, it is a super-fast and scalable no-SQL data layer with a bunch of useful in-memory data structures. In some instances, however, it would be great to have schema support for the data in a Redis database. Imagine a case in which we have several different Microservices reading and writing customer profile data into Redis. It would save time and reduce bug count if, for example, Redis would validate that the customer age key is a positive number.

One of the best “hidden gems” in Redis is the Module APIs which allows adding new commands and behaviors to Redis. Redis comes with a C API which is really easy-to-use (it even helps you avoid memory leaks) so even a part-time developer like me is able to create one. In this article we will look at a module that can add data validation rules to Redis as well as implement a simple Table data type. The source code for this article is located at https://github.com/nirmash/redis-schema.

The “schema” Redis module

The Redis module in this article is comprised of three components:

1. Rules — The module allows developers to associate data validation rules with Redis Key name patterns. It supports string length, number range, RegEx, Enum (list of values allowed) and Picklist (name of a Redis set containing allowed values) validation rules. Field patters are evaluated left to right (e.g., a string rule that has the name_ pattern and defines max string length at 10 will apply to a Redis key called name_1 but not name1).
2. Tables — The module can be used to create a table rule. A table rule defines a table name and a list of individual rules that act as columns. In a table, field names are strictly evaluated (e.g., a table that has a firstName field will not accept data for a field called firstName1).
3. Lua scripts — Redis supports running server-side Lua Scripts. The module adds the ability to use Lua scripts as a way to implement queries or further validation on data.

All metadata (e.g,. the data validation rules or scripts) is saved in Redis keys. Redis commands were added to register rules and “upsert” values into the database. The GitHub repo includes the source code for the module and the files needed to run it in a Docker environment.

The rest of the article walks through building and launching the module in a Docker container, loading some test records and adding a Lua script to query them.

Using this sample requires the following

1. Install Git
2. Install Docker
3. Install Docker Compose

Getting the environment up and running

We will start by cloning the Git repo. In a terminal window, type:
git clone https://github.com/nirmash/redis-schema.git

Once done, the files will be under the redis-schema folder.
To start up the environment, type the following into the command window.
cd redis-schema/
To initiate the containers, use launch.bat (Windows CMD) or ./launch.sh (MacOS \ Linux). The script pulls and builds 2 containers, one running a Redis server with the schema module loaded (redis-schema_redis_1) and another container with Lua installed and some test scripts to help demonstrate the module features.
./launch.sh

Note: The docker-compose.yaml manifest maps the Redis port (6379) from the Redis container to your local host. If you have the Redis client (redis-cli) installed on your local host you can connect to the Redis instance running on the redis-schema_redis_1 container. The rest of this article assumes Redis client is not installed on your local host and leverages the redis-schema_lua_debugger_1container to showcase the module capabilities.

Connect to the Lua debugger container

We will use the redis-schema_lua_debugger_1 as our environment. To get started, open a terminal window and type:
docker exec -it redis-schema_lua_debugger_1 bash

We can now connect to Redis and check that the schema module is up and running.
redis-cli -h 172.28.1.4 -p 6379

Note: Since we are using docker compose, the Redis server host is exposed on 172.28.1.4 IP address listening on port 6379.

Verify that the schema module is loaded by typing schema.help.

We will create a couple of schema rules for Redis keys and Redis lists. Let’s imaging that we want to store a list of Vehicle Identification Numbers (VIN) in our Redis database. VIN numbers follow a strict format (upper case characters except for I, O, or Q with the last 6 characters being numeric and overall length of 17 characters). We will validate that a string meets the VIN format by using a RegEx rule (curly braces escaped with ‘\’).
schema.regex_rule vin_ \b[(A-H|J-N|P|R-Z|0-9)]\{17\}\b

To execute validation rules on incoming data, the module includes a key_upsert command. This command takes a Redis command as its first parameter (e.g. set, sadd) a field name and the value(s).
Inserting a VIN (valid and invalid) into a Redis set will look like:

A simpler case would be storing model years for the cars in the list. We would want to make sure the value is reasonable (let’s say 1940–2023). To do that we would us a number schema rule:
schema.number_rule model_year_ 1940 2023

Setting the value for a model_year_valid and invalid) key looks like:

In some cases, we would want to store our data in a tabular form. Let’s imagine we want to store a list of contacts in Redis and be able to filter and retrieve the data. We will start by creating the fields in our table and associate them with a new table we name contacts.
schema.string_rule firstName 20
schema.string_rule lastName 20
schema.number_rule age 0 150
schema.table_rule contacts firstName lastName age

Since Redis doesn’t have a notion of tables we are storing the table data in Redis hashsets. The module is adding an Id key to each table row and increments the number of records in a separate counter key. Let's add some records to the contacts table and see how they are stored in the Redis database.

The module includes the schema.upsert_row command. This command takes a row index as the first parameter (-1 for a new row) followed by the table name and then a list of key value pairs (field name and value). Values are checked against the field rules (e.g., a contact age can't be larger than 150).
schema.upsert_row -1 contacts firstName john lastName doe age 25

Schema.upsert_row creates a hashset called contacts_1 (as it is the first record in the table), and a key called __*_schema_table_pk_contacts that stores the latest row index. If, for example, we want to update the age value in the record we just added we can use schema.upsert_row with 1 as the first parameter.
schema.upsert_row 1 contacts age 35

Let’s insert some more data into our table so we can query it later.
schema.upsert_row -1 contacts firstName john lastName doe age 25 schema.upsert_row -1 contacts firstName jane lastName doe age 30 schema.upsert_row -1 contacts firstName alex lastName hamilton age 45
schema.upsert_row -1 contacts firstName george lastName washington age 50 schema.upsert_row -1 contacts firstName ben lastName smith age 11

We can see the list of hashset keys that were added to Redis below:

Create and debug a Lua query script

Now that we have a table with some data, we need a way to query it. We may also want to run some more business logic that is specific to our data model. To do that, the module includes a way to register and execute Lua Scripts. Let’s look at a script that returns all the data in the contacts table.

Note: The module passes arguments to Lua scripts through the KEYS Lua array and Redis provides the redisobject for scripts to interact with the Redis database that initiated them.

Scripts are registered by using the schema.register_query_lua command. This command takes a script name and Lua source code. The redis-schema_lua_debugger container includes a simple tool to load Lua scripts from files.
To use the tool, open a new terminal window and navigate to the src folder.
docker exec -it redis-schema_lua_debugger_1 bash
cd src
Load the select all script for our table. The lua_publish.py script takes the name of the Lua file we want to load.
python3 lua_publish.py select_all.lua

To use the Lua script we need to use the schema.execute_query_lua command. To select all the data in the contacts table we would:
schema.execute_query_lua select_all.lua contacts

Note: The lua_publish.py uses the name of the Lua file as the name of the script that we called with the schema.execute_query_lua command.

To make developing the Lua scripts easier, we can use the Lua debugger that is built into Redis. We will debug select_all.lua by using the below command:
redis-cli -h 172.28.1.4 -p 6379 --ldb --eval select_all.lua contacts

Note: The Lua debugger expect the actual Lua file with the source code, make sure you execute the debugger from the folder that includes the Lua files.

Conclusion

This article introduced the idea of using Redis Modules to create schema validation rules and leverages the built-in Lua scripting support in Redis to add server-side logic to your Redis backend. Take a look at the code in the GitHub repo.