Ruby Mutex Mayhem

Introduction

A lot of the time here at Cloud 66 we run into race conditions where multiple threads/processes want to act on external resources. The nature of our business means we have a dearth of external resources who states we can not accurately predict at all times meaning we have to code with that in mind (ie. defensively).

But in general, it is pretty common for large or complex applications to require some form of synchronisation when you can not get away with non-blocking algorithms (which is most of us most of the time)

Additional to the above we have many servers (each with many processes) that are responsible for acting on these external resources. This means that we need some form of synchronisation that can run across process/server boundaries (ie. Ruby’s own mutex/file mutex) wouldn’t cut it!

We, therefore, created a RedisMutex class that we could invoke and use our application to ensure that we have consistent synchronisation (where we need it) available everywhere.

In this post, I’ll talk through the way that we implemented the code — and feel free to use it or reach out if you have any feedback or comments. Please note that you should understand the code you’re running and using it is at your own risk; Cloud 66 can not be held liable.

It goes without saying that you’ll need a Redis Server to use this Mutex — but I still said it just in case!

Implementation

  1. This implementation relies on the fact that LUA scripts run against Redis instances are executed atomically. This means that if multiple processes try and execute a script against Redis we know that their execution will not be interleaved.
  2. We rely in Redis to provide us with key expiration in the case that the process responsible did not release the key itself for some reason. This means we need to be careful about the max lock times we set, and also that if the key is released in this way then other processes will continue as if they have exclusive access. This works in our implementation, but please be aware of this should you wish to use this.
  3. $redis is a global variable that encapsulates a connection to your Redis instance. This can be any Redis connection.

Without further ado; See below an implementation of our RedisMutex

# cross-process/cross-server mutex 
class RedisMutex
attr_accessor :global_scope, 
:max_lock_time,
:recheck_frequency
LOCK_ACQUIRER = "return redis.call('setnx', KEYS[1], 1) == 1 and redis.call('expire', KEYS[1], KEYS[2]) and 1 or 0" 
def initialize(global_scope, max_lock_time, recheck_frequency: 1) 
 # the global scope of this mutex (i.e "resource") 
@global_scope = global_scope
 # max time in seconds to hold the mutex 
 # (in case of greedy deadlock) @max_lock_time = max_lock_time 
 # recheck frequency how often to check if the mutex is 
 # released when blocked 
 @recheck_frequency = recheck_frequency 
end 

def synchronise(local_scope = :global, &block)
 # get the lock 
 acquire(local_scope) 
 begin 

# execute the actions
   return block.call

ensure
# release the lock
   release(local_scope) 
  end 
end 
private 

# attempt to acquire the lock

def acquire(local_scope = :global)
  # construct the mutex key; the local scope

# of this mutex (i.e "resource_id")
  mutex_key = "#{@global_scope}. #{local_scope}" 

  # while statement will either get the lock and

# set the expiry on the lock or do neither and return 0
  while $redis.eval(LOCK_ACQUIRER, [mutex_key, @max_lock_time]) != 1 do 
  # wait and try again (until we can get in) 

sleep(@recheck_frequency)
 end 
end 

# release the lock 
def release(local_scope = :global) 
  # return value indicating whether the lock was currently held 
  mutex_key = "#{@global_scope}.#{local_scope}" 
return $redis.del(mutex_key) == 1 
 end 
end

How to use it

Now, see the simple usage sample below.

MY_MUTEX = RedisMutex.new('server_access', 10.seconds)
...
MY_MUTEX.synchronise(server.id) do 
# do some stuff here that needs to be synchronised
# for this resource (accross all application instances)
end

Conclusion

In this blog post we dipped our toes into a multithreading/cross process or server synchronization solution. In a forthcoming blog I’ll look at some other Ruby/Rails multithreading tricks we have use ourselves.

Until then, happy coding!


Originally published at blog.cloud66.com on April 13, 2017.

Show your support

Clapping shows how much you appreciated Cloud 66’s story.