Unlocking Django’s potential: filtering for orphaned records
One of the many popular Django add-ons is the django-filters package, which most Django developers will be familiar with due to its extensive use within the Django REST framework package. I’ve also previously written about this add-on in my article about django-tables as well as my other article about filters in CBVs.
This incredible add-on comes with a plethora of extremely useful features out-of-the-box, to let us developers quickly and easily add filters to any view or API, without having to re-implement the same tedious boilerplate filtering code on all our views. As a developer who has had to provide filtering capabilities across many different views on a web apps, I can safely say this add-on has been a must-have.
For all its capabilities, however, I did recently encounter a situation which django-filters did not provide out-of-the-box: given two models in a many-to-many relationship, I had wanted to provide users with an option to filter for any orphaned records — in order words, records of one model which did not have any related records in the other model.
Our Models
Before diving into the solution, let’s set the stage with some example models which I’ll be referencing from here on out, and better explain the situation.
For this article, I’ll use the simple example of tracking Books and Categories, connected by a multi-to-multi relationship. A book can belong to zero or more Categories, and a category can have zero or more books assigned. Here is what our model definitions look like:
from django.db import models
class Author(models.Model):
first_name = models.CharField(max_length=64)
last_name = models.CharField(max_length=64)
def __str__(self):
return f"{self.first_name} {self.last_name}"
class Category(models.Model):
name = models.CharField(max_length=64)
def __str__(self):
return self.name
class Book(models.Model):
title = models.CharField(max_length=128)
author = models.ForeignKey(Author, on_delete=models.CASCADE)
categories = models.ManyToManyField(Category, related_name='books')
published_date = models.DateField(null=True, blank=True)
pages = models.PositiveIntegerField(null=True, blank=True)W then create a view for the Book model, which will list all our Book records and provide the user with some basic filters. To accomplish this, our CBV inherits from FilterView and incorporates a standard FilterSet class which defines which Book fields to allow the user to filter on:
from django_filters.filterset import FilterSet
from django_filters.views import FilterView
from .models import Book
class BookFilter(FilterSet):
class Meta:
model = Book
fields = ['author', 'categories']
class BookListView(FilterView):
template_name = 'books.html'
model = Book
context_object_name = 'books'
filterset_class = BookFilterFinally, using the Django shell, we can populate our app with a few records so we have some data to work with and filter on:
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from playground.models import Author, Book, Category
>>> fiction = Category.objects.create(name='Fiction')
>>> fantasy = Category.objects.create(name='Fantasy')
>>> ya = Category.objects.create(name='YA')
>>> adventure = Category.objects.create(name='Adventure')
>>> scary = Category.objects.create(name='Scary')
>>>
>>> wells = Author.objects.create(first_name='H. G.', last_name='Wells')
>>>
>>> time_machine = Book.objects.create(title='The Time Machine', author=wells)
>>> invisible_man = Book.objects.create(title='The Invisible Man', author=wells)
>>> time_machine.categories.add(fiction, fantasy)
>>> invisible_man.categories.add(fiction, scary)After adding a few more records, here is what our page now visually looks like with populated data and filters:
Librarians can now use our new view to browse a list of book records stored in the app, and quickly narrow down the results based on the author and categories filter. Unfortunately, there is one key bit of functionality missing: what about when our librarian needs a list of uncategorized books so they know which books still need to be assigned to categories?
In other words, our users require the ability to filter for all books with no category assigned. Also known as orphaned books.
Adding a new option to the filter field
By default, if a user does not select any categories from the “categories” filter, no filtering will be applied on the field, and all records will be shown. This isn’t the behaviour our librarian wants — they need the ability to see only the Book records which have zero categories assigned.
To accomplish this, we’ll first need to insert a new entry in the categories filter multi dropdown that the user can select when filtering. This will indicate to our FilterSet class that the user wants to be shown all Book records without any categories. In order to do this, we’ll manually instantiate the MultipleChoiceFilter class and populate it by hand with the entries we want in the FilterSet’s __init__method. We’ll also append a new entry for “No Categories” with a special value of -1 to denote our special case:
class BookFilter(django_filters.FilterSet):
categories = django_filters.MultipleChoiceFilter(choices=[])
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# Populate the choices for the categories filter, add a custom "No categories" option
queryset = Category.objects.all()
custom_choices = [(obj.id, obj.name) for obj in queryset]
custom_choices.append((-1, "No Categories"))
self.filters['categories'].field.choices = custom_choicesAfter refreshing our browser, we can now see the categories dropdown has a new entry, “No categories” at the bottom of the list:
Now that users have a new option to choose from, we need to make this option actually do something when they select it.
Defining custom filtering logic
Next, we need to implement the filtering logic. This is accomplished by adding the method kwarg to the MultipleChoiceFilter constructor and defining the referenced method, for example with the filter_categories method below:
class BookFilter(django_filters.FilterSet):
categories = django_filters.MultipleChoiceFilter(choices=[], method='filter_categories')
def filter_categories(self, qs, name, value):
# TODO: add filtering logic here
passInside our new method, we need to implement basic filtering for the common use case of a user selecting multiple categories — this can easily be done using the __in queryset lookup below:
qs.filter(categories__in=value).distinct()This provides the same functionality as the ModelMultipleChoiceFilter we got by default previously — so we need to supplement it with our custom logic to filter for only the records without any categories when the user picks the “no categories” option (i.e. value of -1). This can easily be implemented using the __isnull queryset lookup:
if '-1' in value:
qs.filter(categories__isnull=True).distinct()We do need to account for the possibility of the user selecting both the “No categories” option in conjunction with other categories, so we’ll need to modify the queryset to utilize Q in order to combine the two resulting querysets together:
if '-1' in value:
qs.filter(Q(categories__isnull=True) | Q(categories__in=value)).distinct()Putting it all together
Here is what our FilterSet now looks like once we put it all these different pieces together:
import django_filters
from .models import Book, Category
from django.db.models import Q
class BookFilter(django_filters.FilterSet):
categories = django_filters.MultipleChoiceFilter(choices=[], method='filter_categories')
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# Populate the choices for the categories filter, add a custom "No categories" option
queryset = Category.objects.all()
custom_choices = [(obj.id, obj.name) for obj in queryset]
custom_choices.append((-1, "No Categories"))
self.filters['categories'].field.choices = custom_choices
def filter_categories(self, qs, name, value):
if '-1' in value:
# Filter for records without any categories
return qs.filter(Q(categories__isnull=True) | Q(categories__in=value)).distinct()
elif value:
# Default behavior for selected categories
return qs.filter(categories__in=value).distinct()
return qs
class Meta:
model = Book
fields = ['author', 'categories']The BookListView CBV code stays exactly the same as it was before!
In summary, our improved FilterSet class overwrites the categories many-to-many filter field with a MultipleChoiceFilter which references a new custom filter method. In the __init__ method we dynamically insert the special “No Categories” option with a magic -1 value which tells our custom filter to look for any orphaned Book records.
Our librarian is now able to get a list of all uncategorized books using our new option in the categories filter:
