Django partial forms: don’t miss some fields!

Django has always been at the core of our product here at Botify, and we’ve learned a lot of things in our 5 years of (ab)using the Framework. In this series we will talk about some of the gotchas and solutions we’ve encountered along the way.

Django form Classes enable the organisation of the logic that takes place when submitting a form. Sometimes you want to use the same logic, hence the same Form class, for treating forms submissions that only send a subset of the data. For instance you may want to hide some fields depending on the user role. Our experience shows that in this case, Django mechanisms can be hazardous. In this post we’ll try to analyse why, and we’ll explore two workarounds for avoiding these pitfalls.

A partial form example

Let’s take a real world example: you’re Alice, an entrepreneur trying to build a pirate business.

You plan to have many ships and employees, but only some employees will have the rank of “captain”. You build an app so people can sign up with their name, but only the authenticated users (= you) are supposed to decide which one will be captains, so you restrict this field access accordingly.

Your form class looks like this:

While your HTML template looks like this:

The important part is the “if user” condition in both places.

Things look alright in the beginning, but as always with a large codebase, they don’t stay that simple for long:

Time flies and your business grows.

You take an intern, Bob, and get him an account on the app so he can edit names and access other sections. But he’s an intern, so he will make mistakes, and it’d be better not make your captains angry.

So you restrict access to the precious “is_captain” field to only you.

You change your template like this:

You now check the user’s name to display the checkbox or not.

Good? Of course not!

With such a small example, you dear reader may suspect that we have to edit the form object too. Actually we should, because if we don’t put the same condition on the PirateForm class:

• Alice will see the checkbox and send its value to the server, where it will be handled ;
• unauthenticated users won’t see the checkbox, and the field is ignored server-side for them, so it has no effect ;
• Bob won’t see the checkbox, but the field is not ignored server-side for him if you didn’t change the condition. In such a case anytime Django will set it to its default value, which is False, even when Bob was editing the notice for a captain. In other words, Bob will silently reset the “is_captain” field to “False” any time he updates a notice. BUG!

It’s very easy to spot the bug here, but when the conditions are more complex, the forms longer, and the codebase has tens of thousands of lines of code, it becomes really easy to overlook the case, and it’s pretty hard to debug it.

Why does Django reset fields it doesn’t receive?

In the example above, “security” comes to mind as a good candidate for explaining the need to filter things in “PirateForm”. You want to control server-side what was sent, since the HTML is editable client-side and is not secure in that regard. There might be some truth in there, but actually you see that it’s fairly easy to introduce a bug that would allow some kind of users to wipe out data they shouldn’t have access to. Also, I described something related to user profiles, but it could have been totally something else, like a condition that depends on the hour of the day, the country of the user, anything.

The thing is when you submit a form, not all fields send their value to the server explicitly. In particular, submitting an unchecked checkbox sends nothing to the server. So when a user un-checks a particular checkbox, and submits a form, the server has to find a way to detect that it needs to switch the corresponding boolean to False. Django solution is to have a developer-provided list of all fields to handle server-side, so when a form is submitted, it knows which fields needs to be handled, and the ones missing in the submission get their blank value, probably because of that stupid HTML that didn’t send them.

Note that the behaviour has effects beyond checkboxes: in the production bug we had at work, we were resetting both a checkbox/boolean and a textarea/text. Both were hidden behind a conditional that would not be truthy in very specific contexts, and the condition on the corresponding Form object was not excluding those fields properly.

There might be other reasons for Django to promote hardcoded fields and duplication of logic, happy to read some if you know them.

How to avoid this pitfall: better code hygiene

You may add a comment on both sides:

Better than nothing, but not very solid. This does not respect Single Point Of Truth principle and as the codebase (and team) grows, these two pieces of code will become desynchronized.

Our current solution is to wrap the condition in a form method:

This is slightly better, since an edit on any side will lead to check what’s the method and see the reason why it’s there in the first place. The SPOT principle is preserved, and the code is safer.

Another solution: patching HTML behaviour

Other web frameworks use an interesting mechanism to workaround this limitation of the HTML spec (Rails, CakePHP, …): they generate an auxiliary hidden field just before the checkbox, with a special value (for instance “0”):

• if the checkbox is left unchecked, the server receives a value of “0” for that form element name ;
• if the checkbox is checked, the server receives two values of “0” and “1” for that form element name ; the HTML spec states that they should be sent in the order they appear in the form, so if the server parse the parameters correctly, it will process it as “1” and things will be OK.

Back to our example, this part of the form could become:

Note that you should set the special value to “false” like I did above, so that Django’s CheckboxInput will consider the value as falsy (see source). You may want to encapsulate this logic in a custom Widget so it is used on all your checkboxes.

Now that this fundamental problem is solved, we can assume we get the exhaustive list of fields sent to the server. Thus if a field is absent, we should ignore it, not reset it to its default value.

Our form becomes:

The form logic doesn’t contain conditions about the user anymore. You might still want to do that from a security perspective, but at least you won’t have data bugs produced by incoherence between your template and your form object. Obviously this behaviour can be extracted to a mixin so it can be reused easily on other forms, since it contains nothing specific to the “PirateForm” class.

Should you want to play with a sample app demonstrating the problem, we have a repository available here: https://github.com/jbbarth/django-partial-forms-example.

In this article we explored a common trap that you may encounter while designing partial forms. You may think that the problem is somehow artificial, but in our long experience with Django, the need to have fields displayed dynamically definitely exists, and for us, it’s a legitimate use case of the forms manipulation we presented above.

For more dynamic forms we tend to rely more on a Single Page Application pattern coupled with a REST API. That way we control the serialisation and the submission logic, so it’s easier to have fine grained control over the elements, and interactions with the persistence layer are more straightforward. We still use the approach above in many places though, since Django forms are so easy to use and wire!

How are you handling theses cases in your Django codebase? Do you have a solution that scales well, and keeps the code clean? Feel free to chime in in the comments!

Interested in joining us? We’re hiring! Don’ t hesitate to shoot me an email if there are no open positions that match your skills, we are always on the lookout for passionate people.