Quick, write me a Redis client

A beautiful protocol makes implementation easy

Photo by Joel Filipe

Redis is a networked, in-memory key-value store with optional durability, supporting different kinds of abstract data structures. Redis can be used to implement various server side architectural patterns.

You interact with Redis using a client/server protocol. Redis’ protocol is beautifully simple, which makes implementing it easy.

This article shows how to quickly implement a minimal, yet fully functional Redis client in Pharo.

The protocol

RESP (REdis Serialization Protocol) is a request/response protocol over a TCP connection. There are only 5 data types: simple strings, errors, integers, bulk strings and arrays. The first character indicates the type.

Simple strings start with a + and run until end of line (CRLF, \r\n). These cannot contain embedded end of lines.

+OK\r\n

Errors are similar but start with a — and end with CRLF.

-ERR unknown command ‘GETT’\r\n

Integers start with with a : and end with CRLF

:1\r\n

Bulk strings can contain anything and are prefixed by a $ and an explicit byte count ending with CRLF. Then follows the actual contents. There is an extra CRLF at the end.

$2\r\nOK\r\n

Arrays are the composite type. They start with a * and an element count ending with CRLF. After that the elements themselves follows.

*2\r\n$3\r\nfoo\r\n$3\r\nbar\r\n

Commands (requests) from the client to the server are sent as an array of bulk strings. For example, the command to set a key to a value is called SET and takes two arguments, the key name and the value, both strings. So, “SET foo 100” will look as follows.

*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\n100\r\n

As a convenience, a simpler entry format is also supported. These inline commands can be written on one line with arguments separated by spaces.

SET foo 100\r\n

Responses can be any of the 5 data types and depend on the command.

The unit tests

In the spirit of test driven development (TDD) we write down our expectations for 2 simple commands. The first is a simple PING command that gives a PONG reply. The second is an ECHO command that answers its argument.

TestCase subclass: #SimpleRedisClientTests
  instanceVariableNames: ‘client’
  classVariableNames: ‘’
  package: ‘SimpleRedisClient’

Each test needs a functional, connected client.

SimpleRedisClientTests>>#setUp
  client := SimpleRedisClient new.
  client open
SimpleRedisClientTests>>#tearDown
  client close

Here is our first test using a simplified inline command.

SimpleRedisClientTests>>#testPing
  self assert: (client executeInline: #PING) equals: #PONG

Our second test uses a general command with arguments.

SimpleRedisClientTests>>#testEcho
  | string |
  string := ‘STR-’ , 99 atRandom asString.
  self assert: (client execute: { #ECHO. string }) equals: string

The client

To connect to a server, our client will need to know a host and port. Once connected, we’ll have a connection representing the binary TCP stream. To work fluently with character based input and output, we’ll need in and out streams.

This is the class definition for the object implementing our Redis client.

Object subclass: #SimpleRedisClient
  instanceVariableNames: ‘host port connection in out’
  classVariableNames: ‘’
  package: ‘SimpleRedisClient’

First we add accessors for host and port, taking into account some useful defaults.

SimpleRedisClient>>#host
  ^ host ifNil: [ host := ‘localhost’ ]
SimpleRedisClient>>#host: string
  host := string
SimpleRedisClient>>#port
  ^ port ifNil: [ port := 6379 ]
SimpleRedisClient>>#port: integer
  port := integer

Opening and closing a client means dealing with the binary TCP socket stream and the characters streams to read from and to write to.

SimpleRedisClient>>#open
  self close.
  connection := ZdcSocketStream 
    openConnectionToHostNamed: self host 
    port: self port.
  in := ZnCharacterReadStream on: connection.
  out := ZnCharacterWriteStream on: connection
SimpleRedisClient>>#close
  connection
    ifNotNil: [ 
      [ connection close ] on: Error do: [ ].
      in := out := connection := nil ]

Calling #close from #open makes it more robust and able to function as a re-open call. The raw TCP socket stream deals in bytes, not characters. The character streams add UTF-8 encoding & decoding.

The character streams wrap the binary socket stream

A normal command execution consists of two phases: writing the command and reading the reply.

SimpleRedisClient>>#execute: commandArgs
  self writeCommand: commandArgs.
  ^ self readReply
SimpleRedisClient>>#executeInline: command
  self writeInlineCommand: command.
  ^ self readReply

What is left is the implementation of the protocol itself: writing commands and reading replies, following the specification.

Writing commands

SimpleRedisClient>>#writeCommand: args
  out nextPut: $*; print: args size; crlf.
  args do: [ :each |
    | string byteCount |
    string := each asString.
    byteCount := out encoder encodedByteCountForString: string.
    out 
      nextPut: $$; print: byteCount; crlf; 
      nextPutAll: string; crlf ].
  out flush
SimpleRedisClient>>#writeInlineCommand: string
  out nextPutAll: string; crlf; flush

Normal commands are arrays of bulk strings. First we write out the element count. Each command argument is converted to a string. The Redis protocol does not specify what encoding to use for strings, that is up to the client. For Redis, it is just a collection of bytes. We chose to use UTF-8, a variable length encoding. The string length after $ is a byte count though, not a character count. Therefor we use the encoder to compute how many bytes are needed to encode the string.

Inline commands are written as a single line.

A command is followed by a #flush to push all data over the wire.

Reading replies

Reading a reply starts by looking at the first character.

SimpleRedisClient>>#readReply
  | first |
  first := in next.
  first = $+ ifTrue: [ ^ in nextLine ].
  first = $: ifTrue: [ ^ in nextLine asInteger ].
  first = $- ifTrue: [ ^ self error: in nextLine ].
  first = $* ifTrue: [ ^ self readArray ].
  first = $$ ifTrue: [ ^ self readBulkString ].
  self error: ‘Unknown reply type’

Simple strings, errors and numbers are handled directly. Arrays and bulk strings are handled by helper methods.

SimpleRedisClient>>#readArray
  | length array |
  length := in nextLine asInteger.
  length = -1 ifTrue: [ ^ nil ].
  array := Array new: length streamContents: [ :elements |
    length timesRepeat: [ elements nextPut: self readReply ] ].
  ^ array

Array lengths of -1 are a special case and are different from an empty array. The correct number of elements are read by recursively invoking #readReply. Arrays can contains elements of different types.

SimpleRedisClient>>#readBulkString
  | byteCount bytes |
  byteCount := in nextLine asInteger.
  byteCount = -1 ifTrue: [ ^ nil ].
  bytes := in wrappedStream next: byteCount.
  in nextLine.
  ^ in encoder decodeBytes: bytes

This last method completes our implementation. Again, a length of -1 is treated as a special case. Since the length is a byte count, we access the #wrappedStream (i.e. the binary socket stream) and read the raw bytes in one block. Then we use the encoder to decode the bytes.

In about 10 significant methods, we have now implemented a minimal but fully functional Redis client.

Photo by Edouard Ki

More unit tests

On top of this deceptively simple protocol, Redis implements a lot of functionality, all of which is accessible with our simple client. Here are a couple of examples, written in the form of additional unit tests.

The details of each command can be found in the online Redis command documentation.

Key-value storage

Here is some basic key-value store usage.

SimpleRedisClientTests>>#testStringGetSetSimple
  | string |
  string := ‘STR-’ , 99 atRandom asString.
  self assert: (client execute: #(DEL foo)) >= 0.
  self assert: (client execute: #(EXISTS foo)) equals: 0.
  self assert: (client execute: #(GET foo)) isNil.
  self assert: (client execute: { #SET. #foo. string }) equals: #OK.
  self assert: (client execute: #(GET foo)) equals: string.
  self assert: (client execute: #(EXISTS foo)) equals: 1.
  self assert: (client execute: #(DEL foo)) > 0.

Counters

The value of a key can be interpreted as an integer counter with atomic operations. Undefined counters start from zero.

SimpleRedisClientTests>>#testSimpleCounter
  client execute: #(DEL mycounter).
  self assert: (client execute: #(INCR mycounter)) equals: 1.
  self assert: (client execute: #(INCR mycounter)) equals: 2.
  self assert: (client execute: #(GET mycounter)) equals: ‘2’.
  self assert: (client execute: #(DECR mycounter)) equals: 1.
  self assert: (client execute: #(INCRBY mycounter 10)) equals: 11.
  client execute: #(DEL mycounter).

Other data structures

Lists, sets, sorted sets and hash tables are some of the supported data structures. You can find more unit tests in the source code distributed (see load instructions at the end).

Queues

Our client is even flexible enough to support some special use cases. Lists can be uses as queues, with a blocking call to wait for new data.

SimpleRedisClientTests>>#testQueueSimple
  | string semaphore |
  string := ‘STR-’ , 99 atRandom asString.
  semaphore := Semaphore new.
  client execute: #(DEL myqueue).
  [
    | anotherClient |
    anotherClient := SimpleRedisClient new.
    anotherClient open.
    semaphore signal.
    "Block waiting for data entering the queue"
    self
      assert: (anotherClient execute: #(BRPOP myqueue 0)) 
      equals: { #myqueue. string }.
    semaphore signal.
    anotherClient close
  ] 
    forkAt: Processor userSchedulingPriority 
    named: ‘testQueueSimple’.
  semaphore wait.
  self assert: (client execute: { #LPUSH. #myqueue. string }) > 0.
  semaphore wait.
  client execute: #(DEL myqueue).

The main thread waits until the forked thread is ready, coordinating using a semaphore. Then it pushes a new item onto the front of the queue.

The forked thread issues a blocking pop on the back of the queue. It will receive and remove the string pushed by the main thread.

Pub/Sub

Another use case is a pub/sub mechanism. Data is published to a channel. Multiple parties can subscribe to this channel. Each subscriber will receive notifications of data published.

SimpleRedisClientTests>>#testPubSubSimple
  | semaphore string |
  string := ‘STR-’ , 99 atRandom asString.
  semaphore := Semaphore new.
  client execute: #(DEL mychannel).
 
  [
    | anotherClient |
    semaphore wait.
    anotherClient := SimpleRedisClient new.
    anotherClient open.
    self assert: (anotherClient execute:
      { #PUBLISH. #mychannel. string }) > 0.
    anotherClient close
  ] 
    forkAt: Processor userBackgroundPriority 
    named: ‘testPubSubSimple’.
  self 
    assert: (client execute: #(SUBSCRIBE mychannel))
    equals: #(subscribe mychannel 1).
  semaphore signal.
  "Block waiting for data distributed over the channel" 
  self 
    assert: client readReply 
    equals: { #message. #mychannel. string }.
  self 
    assert: (client execute: #(UNSUBSCRIBE mychannel)) 
    equals: #(unsubscribe mychannel 0).
  client execute: #(DEL mychannel).

Here the main thread blocks on reading an incoming notification after subscribing to the channel.

The forked thread publishes a string to the channel. The main thread will receive this string as a message from the channel.

Conclusion

It is a clever move of the Redis designers to offer such an elegant and simple protocol. It makes it easy to implement new clients, which leads to more users for their software.

For a dynamic, interactive language like Pharo it is important to be able to interact with as many services and servers as possible. Luckily writing new clients can be easy as well as fun.

Source Code

The source code for SimpleRedisClient and SimpleRedisClientTests can be found on GitHub under the URL https://github.com/svenvc/SimpleRedisClient.

To load the code in Pharo 6.1 or later, open World > Tools > Iceberg, click + to add a repository, select the option Clone From github.com and enter svenvc as owner name and SimpleRedisClient as project name. With the new repository selected, select Metacello > Install baseline of SimpleRedisClient.