Redesigning an autocomplete for Django

Django is the ten years old web framework, with an active community and thousands production quality 3d party packages available. Several packages implement autocomplete for the Django forms. Why someone would like to discard all of them, and write something new? Here is the story.


I’m working on the django-material project, with the goal is to provide the strong foundation on the Google Material Design for a data entry application. The project is based on MaterializeCSS library underneath. That’s one of the best CSS frameworks for the Material Design if you care about form usage usability with all that tiny things like keyboard navigation.

So, recently I stumbled upon the question, what options do I have to enable autocomplete for a form fields? Such common functionality should not be implemented from scratch. In an ideal case, I should reuse existing code as much as possible.

Client side

I started with exploring the client side: The MaterializeCSS itself has no support for AJAX requests in the autocomplete. That makes it unusable for big datasets. That’s well-known problem, and there are two projects on Github that fix it.

Both of them are started from the built-in MaterializeCSS autocomplete code. Both looks not very popular. When the original repository has over 20k stars, the icefox0801 has 40 and dellert just 2. icefox0801 wasn’t active for last 3 month and over 13 issues stay open. As opposite, deller was active recently but seems project didn’t receive any community attention. The biggest problem that both diverge from the baseline a lot. There is no chance that one of them would be merged as is into the MaterializeCSS, and if there any issues that were already fixed in the core library they are still present in the forks.

Server side

DjangoPackages website has the big list of the packages that implement autocomplete functionality for Django. Lookup for recently updated, and popular, leaves only three of them: django-autocomplete-light, django-ajax-selects and django-select2.

For the server side, the simplicity and the amount of required for the integration are the important criteria for the winner selection.

  • django-autocomplete-light is the oldest package, first released almost five years ago. It follows the simple approach — each autocomplete field need to have corresponding autocomplete view that needs to be listed in URLConfig to be available to return data for the suggestions.
  • django-ajax-selects makes one step further, by introducing abstraction over autocomplete views — autocomplete channels. And by that, eliminating requirements to registry channels separately in the URLConfig. The single autocomplete view, uses the global channel registry to match the request to the appropriate channel.
  • django-select2 tries to beat code duplication, and on a user request, keeps a reference to the widget, that need to autocomplete query in the global cache under a secret key, The browser, using this key, can perform a request to the universal autocomplete view. View deserialize the widget from the cache and uses widget’s queryset to create a response.

django-autocomplete-light and django-select2 use jQuery Select2 library for the client side. django-ajax-selects uses jQuery UI library with an additional 250 lines of custom javascript.

That’s not very exciting review results. It looks dull to duplicate forms querysets and permissions check in the separate views/channels. But the second idea with the widget and queryset serialization/deserialization to the cache looks is, at least, controversial.

There is still a problem with the client side: jQuery Select2 inserts too many elements in DOM that require over 700+ lines of CSS to customize to follow the Material Design guidelines.

But jQuery UI autocomplete is lightweight and looks more promising for the customization to the Material Design look.

Solution

So, what other options do we have? Beyond jQuery UI, there is another popular and lightweight autocomplete solution devbridge/jQuery-Autocomplete Both have very similar API, but devbridge has a bit more functionality. Both use existing text input, so only custom styles for the autocomplete dropdown need to be provided.

That’s was the last step in my investigation process. When designing more complex things, it worth to look how other web frameworks solves this. Rails, Laravel and ASP.NET MVC open source code could contain very inspiring ideas. But switching between the languages could sometimes be mind blowing

To start coding, I selected the devbridge and got working client side in a few lines of JavaScript and CSS code. Solution simplicity reassures that in future it would not leads to new problems, and would be easy to support.

Many2Many fields require a bit more code, to integrate it with MaterializeCSS Chips control and hidden select.


What about server side? It’s pretty clear, if we don’t want to duplicate queries and permission checking over the views, and don’t want to have too much new URLConfig entries we should request autocomplete data from the same view, and the same URL as we got the original HTML page

How could an autocomplete request be made, be easily distinguished on a view side? I started with an idea just do the HTTP GET, just with a custom header indicates that autocomplete data is requested. But any HTTP GET could easy clash with original HTML response cache. HTTP POST could clash with form processing.

What about HTTP OPTIONS? Isn’t this controversial solution also? There are several different opinions over the Internet, should HTTP OPTIONS ever be used. Although many REST API implementations use OPTIONS to self-describe API Endpoint, the usual HTML-driven websites just ignore OPTIONS requests at all.

Let’s look at the specification.

The OPTIONS method represents a request for information about the communication options available on the request/response chain identified by the Request-URI. This method allows the client to determine the options and/or requirements associated with a resource, or the capabilities of a server, without implying a resource action or initiating a resource retrieval.

If we read this literally, OPTIONS requests could be considered the only right way, to describe what values for a field are accepted by the server. OPTIONS request has a body, and corresponding Content-Type and Content-Encoding headers sent with it. So same as with HTTP GET, we can send query parameters with typed value to autocomplete suggestions. OPTIONS requests are not cached, so it’s also could be considered okay. If required, we can create separate caches for them.

And the alternatives which are the code duplication, or widget serialization look even worse than the usage of non-community accepted but standard HTTP request.

Accidentally here, I realized that at this step I almost have the working solution. It’s easy enough to integrate OPTIONS request to the idiomatic django functional based view

def function_based_view(request):
form = MyForm(request.POST or None)
    if request.method == 'OPTIONS':
query = QueryDict(request.body, encoding=request.encoding)
return get_autocomplete_suggestions(form, query)

elif form.is_valid():
form.save()
return redirect('success')
return render(request, 'form.html', {'form': form}

And even simpler for class based views, where the Django default implementation already handles HTTP OPTIONS separately. So we can handle this in a straightforward and separate mixin class.

class FormAjaxCompleteMixin(object):
def options(self, request, *args, **kwargs):
form = self.get_form()
query = QueryDict(request.body, encoding=request.encoding)
return get_autocomplete_suggestions(form, query)

Conclusion

Although the result code diverged far away from the existing alternatives, the Django community packages are the still important part of the development process. They help to evaluate possible solutions ways quickly and spot the major pros and cons of each one. Even after the ten years history of the django community, there is always space for improvements.

The result autocomplete could be seen in action on the django-material demo website, and is going to be available in the next PRO release.