Elements dives under the hood
Django Under the Hood is a Django conference held in Amsterdam. The talks are oriented on projects commonly used alongside Django and how Django itself works “under the hood”, so developers can get a full insight on the tools they use. This is the second installment of the conference that started in 2014. Of course we at Elements couldn’t miss it as we are always eager to know more about Python/Django, so we went there with a group and this is what we learned there.
Django Under the Hood
As mentioned before, the orientation of each of the talks was to let you know how stuff works on the inside, and with that in mind, the lineup was filled with core developers of the different projects. There is no better way to get insights of a project than to get someone that knows the creepy dark rooms exactly.
Open Source software
Let’s start with the basics. One of the most important things I learned during the conference was that you can convince a judge that contributing to open source projects is considered valid community service! It doesn’t mean you have to go and commit a crime so you can contribute, but it’s always good to know :)
Now let’s get serious, there were some interesting points I could take with me thanks to the conference, one of them was knowing the weak points of the projects, almost all speakers pointed out in their talks what parts of their projects need some love, and they did it with lots of detail, so at the end of each talk you were filled with lots of will to contribute to all these amazing projects.
Another thing that I noticed is the great amount of self-criticism some core developers have on themselves and the projects they work on, they know what the problems are and they recognize mistakes made in the past. I think this is the first step into improvement, and I’m glad it’s in practice in the development community of the tools I use every day.
One thing that should be the banner of open source is that you don’t need to be an elite hacker to contribute to open source projects, heck, you don’t even have to be a developer to do it, there’s many ways to help, so go and do it, don’t wait for community service to be assigned to you by a judge!
Talks
Since the topics on the conference were broad, it’s difficult to summarize the whole conference content in a couple of sentences, so what I’m going to do next is to try to briefly describe each of them.
Django HTTP
Jacob Kaplan-Moss was the one in charge of enlightening us in what exactly happens when a request is made to a Django application, precisely every step the request takes since it enters the WSGI Server until it completes the response. Let’s remember WSGI is an interface for web servers and web applications in Python to talk to each other, and of course Django implements that interface.
Thanks to Jacob’s For dummies talk, we learned why tools like WhiteNoise work as a WSGI Middleware and the advantages on doing that against doing it inside Django (clue: by doing that you skip all the layers every request has to go when Django processes it). In fact, you might want to evaluate all your custom Django middlewares and think about implementing them as WSGI middlewares.
What’s next: Django channels will put Django beyond the request/response cycle and more welcoming towards technologies like HTTP2 and WebSockets, as well as allowing some pretty cool stuff like background processing without the need of Celery! I strongly encourage you to take a look at it as it will be part of Django core in the (hopefully near) future.
Django Admin
After getting insight of the future of Django. Ola Sitarska came to talk about one of the ancient parts of the project, the (oh-so-awesome) admin interface, for all of you that haven’t had the joy of working with Django, it gives you an auto-generated full interface where you can manage your entire website with little to no effort. The not-so-good part is that the Admin is one of the things that hasn’t changed a lot over the years, what’s amazing is that it has been barely touched in 8 years and it just keeps working across versions, talk about stability huh?
Ola guided us through the public APIs you can use to make the admin adapt to your needs and obey your commands, but she also told us about some dark parts of the code that you can also take advantage from.
After that came a really good explanation on why it is so difficult to do a full revamp. The short answer to that is that apart from the complexity of it, there are so many third parties heavily using and tweaking the admin interface that would suffer massively from this.
Django-CMS
After a short break to let the knowledge sink in and in the last talk of the day, Iacopo Spalletti came to us to discuss about Django CMS and how it takes advantage (and abuses a little) of Django’s flexibility. We are talking about a different beast of what you might think when you see the acronym CMS, in general, a CMS is a whole system that manages your entire site (think WordPress, Joomla, Drupal, etc.), and if you want it to be differently, you need to take out your wand and start doing some sorcery with things like routing in the web server side, or using subdomains, and the one that causes more nightmares, trying to touch the CMS code (sorry for the nightmares).
There comes Django-CMS, it is designed to be just a Django application, so it will live happily ever after alongside all other Django applications you have in your project, and it makes it easy to interact with them also, so for all you CMS lovers, go take a look, I’m sure you’ll be surprised.
But we are not here to talk about what it does, we are here to see under the hood, so let’s dive into it. As you may imagine, Django CMS extends the Django admin interface heavily, in fact, almost a third of the code base of the project is dedicated to extending Django’s admin interface, remember when Ola told us it’s quite a challenge to change things in the admin, well, you know who to blame now.
To make it easy to use the CMS, you will need to offer to the end user a simple way to modify even the tiniest bit in the site.
Enter the Django ORM
Leveraging the power of the ORM, Iacopo explained to us how Django-CMS created a full dynamic multilanguage system to handle whatever content you want to add to your site and it also makes it easy to create plugins, the not so good part is that this brings heavy load to your database, that’s why Django-CMS relies on caching to keep things under control.
Keynote
Russell Keith-Magee was in charge of the Keynote, he told the story of Django in a particular manner, using the commit messages throughout time. He also discussed about the technical challenges the Django project has right now, but he mentioned the biggest issue Django (and practically any OSS community) is the inclusion of new contributors, specially when it’s a project as mature as Django where all small tasks are already implemented and all that’s left is huge new functionalities, those are commonly implemented by core contributors, but every project needs new blood.
But when the community grows, also coexistence starts to play a major role, so Russel talked about the code of conduct and how Django is being used as an example to other communities to cope with this kind of problem.
Expressions
After Russell’s inspirational talk, we dove again into Django internals, where Josh Smeaton talked about the new Expressions API since Django 1.8, and guided us in how it works and the easiness of implementing new ones.
Expressions are used to make computations in the database level, we’ve had the F() expressions for quite some time now, but it didn’t have a public API to extend it, and sometimes you would need to resort to .raw() or .extra() to achieve what you wanted, with this new API, it’s almost guaranteed you won’t need to go outside of the regular ORM API to make the weirdest query you can think of, and even if it’s not possible, it’ll be super easy to just add a new expression that handles the sql for you, so give the new Expressions a try!
Twisted & Django
After a traditional Dutch lunch, and some fresh air, we were ready to keep receiving knowledge, and Amber Brown was ready to teach us something about Twisted and how you can make it play alongside Django to make some pretty cool stuff. Twisted is a non-blocking event-driven networking engine written in Python, and with few modifications in the Django ORM, Amber could manage to get Django running in an async manner.
You might remember some lines ago when we discussed about Django channels?, by using the combination of twisted and Django channels, you can ditch WSGI servers and just run your Django project inside twisted, and voilà, async Django!
Static Files in Django
After Amber’s talk, it was James Aylett’s turn to guide us deep into the Django asset pipeline, he explained what Django does good, and what could (greatly) improve about the current static file handling in Django. In his slides he kept pointing out all relevant Django issues related with the assets pipeline, so feel free to go to his website, take one and dive into it, who knows, maybe even fix it :)
The latter part of the talk was dedicated at taking a look at how different projects handle the pipeline, and comparing it to how Django does it, so we could have a glimpse on how Ruby on Rails does it, and also how the JavaScript world is working on that.
But, how can we have a future-proof asset pipeline? James’ bet is to be compatible with the way the node.js community is doing it, since they are the ones with more activity in the field right now, the difficult part is keep up with the instability of that development, where every month comes a new thing to replace the old one.
Django Security
Next in line was Florian Apolloner to discuss about security in the Django level, he made sure that all of us knew he wasn’t a security expert and that his talk was mainly focused on how to prevent common attacks to the Django application, nothing more, so for instance, attacks to SSL wouldn’t be covered in the talk because it’s outside Django’s scope, and the things you need to do to ensure you are as protected as possible.
He mentioned one of the first things you could check is the OWASP Top 10 list and see how Django helps you in each point of the list. He dedicated some time to the most common security issues and how to deal with them, I’ll present the very short summary of those next:
- SQL Injections: Django protects you, so as long as you don’t do nasty things with raw() or extra(), it has you covered
- Authentication and Sessions: Don’t try to implement your own, please use django’s builtin functions since it does lots to protect you.
- Cross Site Scripting (XSS): Django’s auto escaping works perfectly for html, but if you are rendering javascript or json in your templates, you need to be careful, use the escapejs filter, or tools like django-argonauts json filter.
- Cross Site Request Forgery: Django does a great work if you use the builtin CSRF functionality, so, well, just use it.
Additional to all of this, you get the security checklist in the manage.py command (manage.py check –deploy) that will list you some things you might have forgotten before deploying to production.
Documentation Systems
Documentation, the nemesis of developers (well, maybe just me), and there was Eric Holscher to try to show us the light.
Eric is the father of ReadTheDocs.org and he talked to us about the Sphinx documentation system, how it works extending reStructuredText, and in turn, how Django and many Python projects use Sphinx to generate the project documentation. He explained that one of the main functionalities of Sphinx is maintaining the application environment and in consequence, a common context to be used when parsing the RST documents, and you can easily extend the platform to adapt it to your needs.
He then proceeded to talk about how Django has an extension of Sphinx where all the documentation is generated in JSON blobs that are processed as context for the templates used in Django Project, in contrast, ReadTheDocs takes the same source and generates the HTML being presented as the documentation in Django ReadTheDocs, with Sphinx you can have as many output formats as you want (or can write the Formatter for it) so the sky is the limit!
That’s all folks!
If you want more information, keep an eye on the DUTH website, in the following days all the talks will be published so you can see all these wonderful people talking about these amazing projects. I encourage you all to look under the hood of your favorite projects so you can have a deeper insight on how they work, and of course contribute in whatever way you can.
See you next time.
(Update November 25th: Opbeat posted videos of the talks on their website.)
Originally published at www.elements.nl on November 13, 2015.
