Django — A Story of Race conditions with get_or_create and unique constraints

Don’t repeat yourself. Ever. Ever. Oh, wait, my bad. Sorry.

Here, I spilled out one of the 101 secret potions we make everyone drink before joining our team! It simply means if you are writing something similar twice, there must be a better way to do that.

Recently while we were writing our Document collection service, under the effects of the above potion. The aim was to assemble a more generic document collection service to collect all kinds of user documents. I will talk about how we, our code and then the sentry got into a weird dilemma during this time and how we solved that.

A critical table in any such service will be a Document table with key fields, namely: user , document_type (AADHAAR_CARD/PAN_CARD/SALARY_SLIP) , document. At first, adding a unique_together constraint for the fields users , and document_type seems logical, keeping in mind that one user should possess only one Aadhaar or PAN Card. But it was not one of the best decisions by us given we will be saving Salary slips, Tax Returns in the same table, which one user can possess in multiple quantities. Confidently we put a validation on the save method of our Django Model like this and went on to sleep peacefully.

def save(self):
  if not self.document_type not in ['AADHAAR_CARD', 'PAN_CARD']:
    if Document.objects.filter(
          user=self.user, 
          document_type=self.document_type
       ).exclude(id=self.id).exists():
      raise ValidationError("Not allowed")
  super().save()

Getting an aadhaar document of a user is a seemingly easy task ahead with something like the following:

user_aadhaar_card = None
try:
  user_aadhaar_card = Document.objects.get(
        user=request.user, 
        document_type="AADHAAR_CARD"
)
except UserDocumentMapping.DoesNotExist:
  user_aadhaar_card = Document.objects.create(
        user=request.user, 
        document_type=AADHAAR_CARD
)

If you ever find yourself writing this, do remember excellent util get_or_create provided by Django, which does a bit more necessity that we generally ignore. Let’s see what get_or_create should be doing under the hood:

try:
  return some_model.get(**queryset), False
except some_model.DoesNotExist:
  try:
    instance = some_model.create(**queryset)
    return instance, True
  except IntegrityError:
    return some_model.get(**queryset), False

This might seem weird repeating the same logic in the except clause, but this comes into the picture when we are working with concurrent requests. Let us try to understand this pictorially.

The execution of the above-mentioned functions for concurrent requests

So, assuming the required unique constraints in their place, your implementation will throw an integrity error, but Django’s get_or_create will catch that and provide a neat response for you.

But as we could not put any unique_contraint above, in our case, it will happily create 2 Addhaar Cards for a single user (UIDAI, code red!) and the next time when I confidently run get(user=user, document_type="AADHAAR_CARD") relying on my poor clean function and Django’s get_or_create it will return me never expected Multiple objects. FML!

Django 2.2 to the rescue; introduced UniqueConstraint , which allows you to apply unique constraints conditionally to your models. In code, something similar to:

class Document(models.Model)
  user = <>
  document_type = <>
  document = <>

  class Meta:
      constraints = [
        models.UniqueConstraint(
          fields=['user', 'document_type'], 
          condition=Q(
            document_type__in=["AADHAAR_CARD", "PAN_CARD"]
          ),
          name='unique_user_unique_document_type'
        )
      ]

And then we live happily. No more sentry alerts, no more users with multiple Aadhaar cards! Phew!

Coming to an end, the moral of the story is to —

Never ever use get_or_create or similar logic with a lookup which is not “unique-constrained” on the database level!