Using dynamic unit tests to build sane Django forms.

Even though we’re drifting towards Front-ends written entirely in JavaScript, Django forms has saved developers countless hours that would’ve been spent writing html by hand or building an API for a JavaScript Front-end to submit data to. It would continue to do so for the foreseeable future because sometimes you don’t want to bother yourself with the extra complexity associated with JavaScript front-end frameworks, You just want a “Plain ol form” that works.

They can also be the source of some annoying “bugs” in your Django application. Especially when you’re Rendering form fields manually.

Everything seemed okay.

Recently, I was working with a fellow developer on a project and i gave him a task to override one of django-allauth’s forms. Specifically, i told him to override the password_reset_key_done.html template to use a shiny Bootstrap template we just got. It’s a form that allows a user reset their password after clicking on a password reset link. It contains two fields one for the password and another to confirm it.

He wrote something like this, committed it and opened a Merge Request.

{% if form %}
  <form method=”POST” action=”{{ action_url }}”>
   {% csrf_token %}
   {{ form.password1|add_class:”form-control” }}
   {{ form.password2|add_class:”form-control” }}
   <input class="btn btn-info" type=”submit” value=”{% trans ‘change password’ %}”/>
  </form>
{% else %}
  <p>{% trans 'Your password is now changed.' %}</p>
{% endif %}

add_class is a custom template filter that adds extra classes to a html element, in this case he used it to add Bootstrap’s “form-control” class to the <input /> tag generated by Django.

It was already late in the night and i was already tired and frustrated from my battle with Webpack earlier in the day.

Everything looked fine, at least all the tests passed and the form looked just like i expected it to be. I was about to click the “Merge” button on Gitlab but something told me to stop and check if the form still functioned as it did before. So i pulled his branch and checked it out on my system.

I filled in the form, and submitted it… Nothing happened, it just came back the same way. I quickly filled it again and hit Enter…Nothing happened again. I tried one more time. Still the form refused to submit. It just came back with no errors whatsoever.

The problem

I knew something was wrong somewhere, but i didn’t even know where to start looking because i didn’t expect this simple change to break anything at all.

“For Goodness sake it, It’s just a new template, how could it possibly break/affect the behaviour of Python. No it’s not possible” i said to myself. I must have done something prior to this change that broke the code without my knowledge. I pursued that path for the next hour, i checked out old commits making sure to include the new template. Even down to the Last month (That was how paranoid i became). Still the form wasn’t submitting.

What the hell is happening!

So i decided to remove the template totally and just allow django-allauth’s default template do it’s thing. The form didn’t submit but i was greeted with some error messages that the password i was entering wasn’t valid. At this point i knew something about the new template wasn’t quite okay because i didn’t see any error messages the form just refused to submit. I decided to take a short nap to clear my head a little and it turned out to be a Good night’s rest till 6:00am in the morning 😃.

The Real problem

From the previous night, I already knew there was a problem, but i didn’t know what it was exactly, but i was sure that i was “going to get it together somehow” (Coldplay’s Up and Up was playing in the background).

I went back to my laptop and tested the Form again carefully filling out the password fields and it submitted! Then i went back to the code and i saw the problem staring right at me.

There were no error messages for the form fields, so the form validation had been failing throughout yesterday and i did not know because my colleague forgot to include the error messages so i couldn’t see them.

This is a very terrible thing, if this kind of Silent bug makes it into production:

1. Users may never report it, so it’s left to the developers to discover it accidentally, since there are no tests to prevent it. You should never rely on users to report errors or strange behaviours in your application. Most people don’t have the time and will just run off to a similar application that offers the same service.
2. You’ll potentially lose a lot of users. Imagine if this was a registration form. The user would storm off and never return to your website again(I’ve done that severally). That’s just bad for business.
3. You’ll also lose a lot of revenue. Just imagine if this form was a checkout form for your online store and users can’t checkout the items in their cart because the form refuses to submit and is “behaving strangely”. Most people would immediately open E-bay or some other alternative and forget about your store. Some might even go as far as telling their friends and family that they shouldn’t use your website because they wasted their time searching and adding lots of items to their cart only for them to get stuck in the checkout process.

From the End-user’s perspective, the form is simply not working (Even though they’re actually entering incorrect values) and it’s not giving them any hint as to why it’s not working.

The most annoying part of all this is that even if you notice a decline in revenue/user-base, You may never know that the problem is from a form without any error messages. Like me, The business may start pursuing other solutions that aren’t even related to the original problem Like Aggressive advertising, Adding new features etc. Hopefully, the developers might eventually discover the problem but by then a lot of money/users would’ve been lost to a “Simple mistake”.

Error tracking software like Sentry, Rollbar, Raygun etc wouldn’t help you track this kind of bugs/errors, because even if Django’s forms.ValidationError’s are Exceptions, They are not uncaught exceptions. They are already handled by Django and it’s expected that you render them on the form.

The Solution

As Humans, we’re bound to forget. In fact, I’d made this mistake several times in the past. So i didn’t blame my Co-worker for this. Other things started to come to my mind, “What if there were other forms like this in the application that i didn’t know of”. We already had over 20 forms in the application and checking them manually was going to take a lot of time and energy.

Even if we had just 2 forms, what would happen as we kept on adding more forms or modifying forms and someone forgot to render the error message of a field in the template, mistakenly deleted one of the error fields or made a typo e.g {{ form.password.errrorrs }}. I quickly discarded the idea of Manual checking in favour of Automated tests.

Even if i wanted to write tests, how was i going to write them? Would i write a test for each form in the application? Though this approach was better, it still suffered from the same problem as the manual method of checking. I have to “check” (Modify the code and add new tests) anytime i create a new form. Which i or my Co-developer could still forget to do.

I needed a solution that could “Dynamically” do all the checking for me without me having to modify my code anytime i add a new form. The solution needed to be smart enough to know when a new form was added in a template and then test all the form fields if they have corresponding error messages included in the template.

The Real solution (Let your Computer work for you )

I knew that i had to write code that generated and injected tests to a TestCase. This kind of technique is called Meta-programming, where a program has the ability to “Write other programs” or “Alter itself”.

The Solution was actually simpler than i expected it to be. Due to the dynamic nature of python it’s very easy to set attributes on an object at Run time with the built in setattr function. This is not the case for Statically compiled languages.

In other words, I Let my Computer work for me, Instead of the other way.

The most important section of this script are the Regular expressions.

I wrote them to be as flexible as possible so they can catch a lot of field and form formats. they can even test for fields that have extended Unpacking. For example when you try to iterate over a field like this:

<ul>
  {% for value in form.field  %}
  <li>{{ value }}
  {% endfor %}
</ul>

The test script also takes Django template filters into account, so fields that have filters (like add_class in my case) would also be matched by the Regular expressions.

But there’s still a broad assumption, Your form must contain the keyword “form” e.g my_form and form_ismine are valid examples that would be matched. It must not contain the keyword “formset”. More about that below.

Sometimes there are certain fields that you may not want to be checked because they don’t have any validation. A good example is form.remember from django-allauth, it can’t raise any error messasge, so there’s no need checking for one. you can just include the fields name in the UNWANTED tuple and have it filtered out.

Also it should be noted that i didn’t take Django formsets into consideration Personally, i try to avoid them as much as possible when using Django. As a rule of thumb, if i find myself needing formsets to manipulate related models, i know it’s probably time to start building an API that can be used by any JavaScript front-end to make the form as interactive as possible.

If you use formsets heavily in your application and you need to test them, you just need to write a new Regular expressions to match them, and add them as a new item in the dictionary returned by parse_templates.

Finally, you have to name the file properly so Django’s Test runner can discover the file and run the tests. I named mine test_error_fields.py

How did the story end?

At the end of the day, when i ran the tests again, the number of tests we had increased from 54 to 102 and 7 of them failed. There were non_field_errors that we failed to include in various forms. That’s not as bad as forgetting individual field errors, But it’s best to make sure all form errors are rendered so users don’t get confused when using your application.

I committed the new file and made it part of our CI process, so each time new code is committed, the tests are run to make sure all our forms are okay.

This is a script i use in production now. For most projects, it should be able to detect all your forms if you have a sane naming convention for them. I’m currently using it for three Django projects without any issues. If you have any suggestions on how this test script can be further improved, please let me know in the comments.

Thanks for reading. Don’t forget to Recommend and Share if you enjoyed this article.

If you’re wondering how i couldn’t get the form to work on the first night, At that moment (because of fatigue), i thought it was a login form and i was trying to login to a password change form. 😆 Also, my username wasn’t up to 8 Characters long as required by the password1 field.

Update: I have written yet another test and i decided to move both of them into a public github repo https://github.com/danidee10/django-dynamic-tests.

The second tests check’s templates for missing Static files, This is very useful when you’re moving static files around and you forget to update the new location in the templates.