Making React and Django play well together — the “hybrid app” model

Aymeric Augustin

Published in

Fractal Ideas

12 min readMay 10, 2018

Please read the updated version of this blog post at fractalideas.com.

This is part two in a three-part series:

Last month I discussed the trade-offs involved in choosing an architecture for integrating React with Django.

I described an alternative between two models:

The “single page app” model: a standalone JavaScript frontend makes API requests to a backend running on another domain.
The “hybrid app” model: the same backend serves HTML pages embedding JavaScript components and API requests.

I promised that I would describe how to implement each model.

Today I’m starting with the “hybrid app” model.

Let’s bootstrap a todolist app!¹

Disclaimer: I’m starting with default project templates and making minimal changes in order to focus on the integration between frontend and backend.

As a consequence, I’m ignoring many best practices for Django and React projects. Keep in mind that I’m describing only one piece of the puzzle and that many variants are possible!

Why build a “hybrid app”?

Here are the main reasons:

Your app needs good SEO (e-commerce, classifieds, portfolios, etc.)
You prefer treating your app as a single unit because that’s more convenient for building new features, running integration tests, and deploying new versions.
You have a team of full-stack developers — perhaps a team of one — and you’d rather avoid the overhead of managing separate backend and frontend projects.
You’re comfortable with building full-stack apps with Django and you want to try a modern JavaScript framework while leveraging your experience.

While the “hybrid app” model feels a bit old-school, it’s good for enhancing the frontend of backend-heavy apps — whether new or pre-existing — without disruption.

Initialization

Since the frontend and the backend are deployed together, it makes sense to maintain them in the same code repository.

Let’s initialize Django and React applications in backend and frontend directories at the root of the repository.²

Django

Start a shell, go to the root of the code repository and bootstrap the backend:

mkdir backend
cd backend
pipenv install django
pipenv shell
django-admin startproject todolist .

For convenience, edit the Django settings as follows. This will make it easier to switch between development and production mode.

# backend/todolist/settings.py

# insert these lines after the definition of BASE_DIR
BACKEND_DIR = BASE_DIR  # rename variable for clarity
FRONTEND_DIR = os.path.abspath(
    os.path.join(BACKEND_DIR, '..', 'frontend'))

# modify the definition of DEBUG and ALLOWED_HOSTS
DEBUG = os.environ.get('DJANGO_ENV') == 'development'
ALLOWED_HOSTS = ['localhost']

Start the development server:

# in the backend directory, after executing pipenv shell
DJANGO_ENV=development ./manage.py migrate
DJANGO_ENV=development ./manage.py runserver

Open http://localhost:8000/ in a browser to confirm that everything is working.

React

Start another shell, go to the root of the code repository and bootstrap the frontend:

npx create-react-app frontend
cd frontend

Start the development server:

# in the frontend directory
yarn start

http://localhost:3000/ opens automatically in a browser.

Starting point

Django and React development servers are now running on http://localhost:8000/ and http://localhost:3000/ but they don’t know anything about each other.

The source tree contains:

.
├── backend
│   ├── Pipfile
│   ├── Pipfile.lock
│   ├── manage.py
│   └── todolist
│       ├── __init__.py
│       ├── settings.py
│       ├── urls.py
│       └── wsgi.py
└── frontend
    ├── README.md
    ├── package.json
    ├── public
    │   ├── favicon.ico
    │   ├── index.html
    │   └── manifest.json
    ├── src
    │   ├── App.css
    │   ├── App.js
    │   ├── App.test.js
    │   ├── index.css
    │   ├── index.js
    │   ├── logo.svg
    │   └── registerServiceWorker.js
    └── yarn.lock

You can confirm this by running tree -I node_modules.

Running yarn build in the frontend directory adds:

.
└── frontend
    └── build
        ├── asset-manifest.json
        ├── favicon.ico
        ├── index.html
        ├── manifest.json
        ├── service-worker.js
        └── static
            ├── css
            │   ├── main.c17080f1.css
            │   └── main.c17080f1.css.map
            ├── js
            │   ├── main.a3b22bcc.js
            │   └── main.a3b22bcc.js.map
            └── media
                └── logo.5d5d9eef.svg

Production setup

Let’s start with the configuration for running the app in production.

Many developers would think about their development setup first.

However, production has more constraints than development, notably security and performance. As a consequence, I prefer starting with the production setup and then building a development setup that provides adequate dev / prod parity.

Serving HTML

In the “hybrid app” model, Django is in charge of serving HTML pages.

Let’s add a view that renders the index.html generated by create-react-app with the Django template engine.³

A catchall URL pattern routes any unrecognized URL with that view. Then the frontend URL router takes over and renders the appropriate page on the client side.

Configure the template engine:

# backend/todolist/settings.py

TEMPLATES = [
    {
        'DIRS': [os.path.join(FRONTEND_DIR, 'build')],
        ...,
    },
]

Create the view in a new module:

# backend/todolist/views.pyfrom django.views.generic import TemplateViewcatchall = TemplateView.as_view(template_name='index.html')

Add a catchall URL pattern to the URLconf:

# backend/todolist/urls.pyfrom django.contrib import admin
from django.urls import path, re_pathfrom . import viewsurlpatterns = [
    path('admin/', admin.site.urls),
    re_path(r'', views.catchall),
]

Build the frontend:

# in the frontend directory
yarn build

Install waitress, a production WSGI server, and start it locally:⁴

# in the backend directory, after executing pipenv shell
pipenv install waitress
waitress-serve todolist.wsgi:application

At this point, you can access the HTML page generated by create-react-app and served by Django at http://localhost:8080/. The page title displayed in the browser tab says “React App”, which confirms that we’re getting the intended HTML page.

However, the page is blank because loading the CSS and JS failed. Let’s fix that.

Serving static files

The “hybrid app” model allows us to take advantage of the staticfiles contrib app and its ecosystem. I find it convenient to serve static files with WhiteNoise and cache them with a CDN for performance.

The changes described below:

Configure django.contrib.staticfiles and WhiteNoise;
Declare that frontend/build/static/ contains static files;
Serve them at /static/ where create-react-app expects them;⁵
Serve the few other files in the frontend/build/ directory at the server root.⁶

# backend/todolist/settings.py

INSTALLED_APPS = [
    ...,
    # before django.contrib.staticfiles
    'whitenoise.runserver_nostatic',
    ...
]

MIDDLEWARE = [
    ...,
    # just after django.middleware.security.SecurityMiddleware
    'whitenoise.middleware.WhiteNoiseMiddleware',
    ...,
]

STATICFILES_DIRS = [os.path.join(FRONTEND_DIR, 'build', 'static')]

STATICFILES_STORAGE = (
    'whitenoise.storage.CompressedManifestStaticFilesStorage')

STATIC_ROOT = os.path.join(BACKEND_DIR, 'static')

STATIC_URL = '/static/'  # already declared in the default settings

WHITENOISE_ROOT = os.path.join(FRONTEND_DIR, 'build', 'root')

This setup requires moving the files to serve at the server root in a root subdirectory. That’s everything in frontend/build except index.html and static.

# in the frontend directory
cd build
mkdir root
mv *.ico *.js *.json root
cd ..

Stop the WSGI server, install WhiteNoise, collect static files, and restart the server:⁷

# in the backend directory, after executing pipenv shell
pipenv install whitenoise
./manage.py collectstatic
waitress-serve todolist.wsgi:application

Now, if you refresh the page at http://localhost:8080/, the create-react-app welcome page loads. Success! Also, the browser tab displays the React icon, which means that serving files like favicon.ico from the server root works.

Deploying

This setup is compatible with any method for deploying Django to production, provided the server that builds the application prior to deploying it is able to build the frontend and the backend. This boils down to installing Node.js and yarn.

A complete build script for the application looks like:

cd frontend# 1. Build the frontend
yarn build# 2. Move files at the build root inside a root subdirectory
mkdir -p build/root
for file in $(ls build | grep -E -v '^(index\.html|static|root)$'); do
    mv "build/$file" build/root;
donecd ..cd backendpipenv shell# 3. Build the backend
./manage.py collectstatic --no-inputcd ..

Development setup

Now that we’re happy with the production setup, let’s build the development setup.

The challenge is to optimize dev / prod parity while preserving all features of the development servers.

Remove all build artifacts to make sure we don’t accidentally rely on them:

# in the backend directory
rm -rf static# in the frontend directory
rm -rf build

If the Django and React development servers aren’t running anymore, start them again with:

# in the backend directory, after executing pipenv shell
DJANGO_ENV=development ./manage.py runserver# in the frontend directory
yarn start

Serving HTML

In production, we’re loading frontend/build/index.html as a template.

In development, it’s available at http://localhost:3000/.

It would be possible to build a Django template loader that loads templates from an URL. That would be optimal in terms of dev / prod parity but I don’t think it’s worth the complexity.

Instead I’m just going to write an alternative catchall view that proxies requests for index.html to the React development server.

Proxying is more appropriate than redirecting: redirecting could change the behavior of the browser and introduce significant differences between dev and prod.

Install Requests:

# in the backend directory
pipenv install requests

Change the views module to:

# backend/todolist/views.pyimport requests
from django import http
from django.conf import settings
from django.template import engines
from django.views.generic import TemplateViewdef catchall_dev(request, upstream='http://localhost:3000'):
    upstream_url = upstream + request.path
    response = requests.get(upstream_url)
    content = engines['django'].from_string(response.text).render()
    return http.HttpResponse(content)catchall_prod = TemplateView.as_view(template_name='index.html')catchall = catchall_dev if settings.DEBUG else catchall_prod

I’m keeping the original implementation in production because it benefits from Django’s caching of compiled templates.

Refresh http://localhost:8000/: while the page is blank, the page title says “React App”, which is good.

However, there’s a stack trace and a HTTP 500 error in the logs of runserver: "GET /static/js/bundle.js HTTP/1.1" 500. Oops, our catchall view is also receiving requests for static files and it crashes when it attempts to compile them as Django templates!

That was our next step anyway, so let’s improve catchall_dev.

Serving static files

Here’s a version that runs only HTML responses through Django’s template engine.

In addition, it avoids buffering static assets in memory. Development builds of the frontend can grow very large because they aren’t optimized for file size like production builds. Using the streaming APIs in requests and Django makes a noticeable difference.

# backend/todolist/views.pyimport requests
from django import http
from django.conf import settings
from django.template import engines
from django.views.generic import TemplateViewdef catchall_dev(request, upstream='http://localhost:3000'):
    upstream_url = upstream + request.path
    response = requests.get(upstream_url, stream=True)
    content_type = response.headers.get('Content-Type')if content_type == 'text/html; charset=UTF-8':
        return http.HttpResponse(
            content=engines['django'].from_string(response.text).render(),
            status=response.status_code,
            reason=response.reason,
        )else:
        return http.StreamingHttpResponse(
            streaming_content=response.iter_content(2 ** 12),
            content_type=content_type,
            status=response.status_code,
            reason=response.reason,
        )catchall_prod = TemplateView.as_view(template_name='index.html')catchall = catchall_dev if settings.DEBUG else catchall_prod

Refresh http://localhost:8000/: the application loads.

Try modifying source files in the frontend. Autoreload works. Hurray!

Optimizing autoreload

At this point, there’s still a couple errors in the logs of runserver:

"GET /sockjs-node/nnn/xxxxxxxx/websocket HTTP/1.1" 400: our proxy doesn’t support WebSocket connections;
"POST /sockjs-node/nnn/xxxxxxxx/xhr_streaming?t=ttttttttttttt HTTP/1.1" 403: our proxy rejects POST requests because they don’t account for Django’s CSRF protection.

SockJS falls back to Server-Sent Events to wait for autoreload events, as the following line in the logs shows: "GET /sockjs-node/nnn/xxxxxxxx/eventsource HTTP/1.1" 200. Server-Sent Events happen to be supported as a side effect of the streaming optimization.

Can we avoid these errors?

The question of dev / prod parity no longer matters here because there’s no autoreload in production.

We can easily add support for POST requests and fix the second error by disabling CSRF protection for the catchall_dev view⁸ and proxying the HTTP method properly.

Unfortunately there’s no trivial way to support WebSocket connections and fix the first error:

proxying them would require setting up Django Channels and writing quite a bit of code;
redirecting them doesn’t work because browsers don’t follow redirects on WebSocket connections;
routing them to the create-react-app dev server isn’t possible at the time of writing; that’s an open issue.

For lack of a better solution, we can detect WebSocket requests and return a HTTP 501 Not Implemented error code.

An error will still be displayed in the console of the web browser and SockJS will fall back to a less efficient mechanism, XHR Streaming, that keeps a Python thread busy.

Here’s the final version:

# backend/todolist/views.pyimport requests
from django import http
from django.conf import settings
from django.template import engines
from django.views.decorators.csrf import csrf_exempt
from django.views.generic import TemplateView@csrf_exempt
def catchall_dev(request, upstream='http://localhost:3000'):
    """
    Proxy HTTP requests to the frontend dev server in development.The implementation is very basic e.g. it doesn't handle HTTP headers."""
    upstream_url = upstream + request.path
    method = request.META['REQUEST_METHOD'].lower()
    response = getattr(requests, method)(upstream_url, stream=True)
    content_type = response.headers.get('Content-Type')if request.META.get('HTTP_UPGRADE', '').lower() == 'websocket':
        return http.HttpResponse(
            content="WebSocket connections aren't supported",
            status=501,
            reason="Not Implemented"
        )elif content_type == 'text/html; charset=UTF-8':
        return http.HttpResponse(
            content=engines['django'].from_string(response.text).render(),
            status=response.status_code,
            reason=response.reason,
        )else:
        return http.StreamingHttpResponse(
            streaming_content=response.iter_content(2 ** 12),
            content_type=content_type,
            status=response.status_code,
            reason=response.reason,
        )catchall_prod = TemplateView.as_view(template_name='index.html')catchall = catchall_dev if settings.DEBUG else catchall_prod

And we’re done!

I didn’t explain all the design choices, so you may be wondering…

Why this design?

The setup I described takes advantage of features available in create-react-app and django.contrib.staticfiles to optimize page load performance and dev / prod parity with very little code and without introducing any additional dependencies.

Caching

Optimizing how browsers cache static assets is a performance-critical requirement.

The most reliable solution consists in:

Inserting a hash of the contents of each file in the name of the file;
Telling the browser that it can cache static files forever.

If the contents of a file changes, then its name changes and the browser loads the new version.

In practice this is more complicated than hashing and renaming files. For example, when CSS references an image, inserting a hash in the name of the image changes the contents of the CSS, which changes its hash. Implementing this behavior involves parsing static assets and understanding dependencies.

Django is able to parse and to modify CSS to handle dependencies. This is implemented in in ManifestStaticFilesStorage, which WhiteNoise’s CompressedManifestStaticFilesStorage builds upon.

Code splitting

Delivering the application gradually with code splitting is another performance-critical requirement.

When code splitting is enabled, a JS loader downloads chunks that define modules and imports them as needed. This is a fairly new requirement: code splitting wasn’t mainstream until two years ago.

Django is unable to parse JS and understand dependencies between JS files. For this reason, the bundler needs to be responsible for inserting hashes in file names.⁹ Generally speaking, since the bundler is responsible for creating an optimized build of frontend assets, it makes sense to let it take care of inserting hashes in file names.

Putting it all together

Regardless of which system performs the hashing, a mapping from original file names to hashed file names is needed in order to substitute the hashed file name automatically whenever the developer references a static asset by its original file name. For example, main.js must be replaced with main.a3b22bcc.js.

The crux of the issue is to transmit this mapping from the bundler, which creates it when it builds the frontend, to the backend, which needs it to reference static files in HTML pages.

In our setup, the mapping is already applied in frontend/build/index.html when Django loads it as a HTML template, so the backend doesn’t need to do anything.

An alternative solution involves dumping the mapping in a JSON file with a webpack plugin, then loading it and applying it with a Django storage engine. webpack-bundle-tracker and django-webpack-loader implement this.

One more thing!

If you’ve been following closely, you noticed that our setup hashes frontend assets twice: webpack hashes them during yarn build and Django hashes them again during ./manage.py collectstatic. This produces files such as backend/static/js/main.a3b22bcc.5c290d7ff561.js.

These double-hashed files are never referenced.

However, generating them makes ./manage.py collectstatic a bit slower than it could be, especially on large apps. It’s possible to optimize this by subclassing CompressedManifestStaticFilesStorage, if you accept relying on private APIs.

Since I know you’re going to ask, here’s an implementation. It’s a fun hack and it works for me. Don’t shout at me it if breaks :-)

# todolist/storage.py

import collections
import re

from django.contrib.staticfiles.storage import  (
    ManifestStaticFilesStorage)
from whitenoise.storage import (
    CompressedStaticFilesMixin, HelpfulExceptionMixin)


class SkipHashedFilesMixin:

    _already_hashed_pattern = re.compile(r'\.[0-9a-f]{8}\.')

    def is_already_hashed(self, path):
        """
        Determine if a file is already hashed by webpack.

        The current implementation is quite lax. Adapt as needed.

        """
        return self._already_hashed_pattern.search(path)

    def post_process(self, paths, dry_run=False, **options):
        """
        Skip files already hashed by webpack.

        """
        if dry_run:
            return

        unhashed_paths = collections.OrderedDict()
        for path, path_info in paths.items():
            if self.is_already_hashed(path):
                yield path, None, False
            else:
                unhashed_paths[path] = path_info

        yield from super().post_process(
            unhashed_paths, dry_run=dry_run, **options)

    def stored_name(self, name):
        if self.is_already_hashed(name):
            return name
        else:
            return super().stored_name(name)


class StaticFilesStorage(
        HelpfulExceptionMixin, CompressedStaticFilesMixin,
        SkipHashedFilesMixin, ManifestStaticFilesStorage):
    """
    Similar to whitenoise.storage.CompressedManifestStaticFilesStorage.

    Add a mixin to avoid hashing files that are already hashed by webpack.

    """

After you create this file, enable the storage engine with:

STATICFILES_STORAGE = 'todolist.storage.StaticFilesStorage'

After you create this file, enable the storage class with:

STATICFILES_STORAGE = 'todolist.storage.StaticFilesStorage'

Footnotes

Implementing the app is left as an exercise for the reader. There are blog posts about building todo list apps on the Internet.
I will be using pipenv and yarn. It’s a matter of personal preference. You can achieve the same result with virtualenv + pip and npm instead.
This makes it possible inject data into the page, if needed, by using Django template syntax in index.html.
I prefer waitress over gunicorn as a zero-configuration, secure WSGI server.
There are many other good solutions; which one you’re choosing doesn’t matter much, as long as you tell create-react-app where the frontend is hosted.
If you’re putting many files in create-react-app’s public folder, which the documentation recommends against, this setup will be inefficient.
Unlike ./manage.py runserver, waitress-serve doesn’t include an auto-reloader. It must be restarted manually to take code changes into account.
I believe that it’s safe to disable CSRF protection on the catchall_dev view because requests are forwarded to the React development server which, at first sight, doesn’t have any server-side state such as a database that an attacker could try to alter. I didn’t investigate further than this. If you’re concerned, perform your own due diligence.
Look for hash in filename patterns in the production webpack configuration of create-react-app if you’re curious.

Originally published at fractalideas.com.

Making React and Django play well together — the “hybrid app” model

Why build a “hybrid app”?

Initialization

Django

React

Starting point

Production setup

Serving HTML

Serving static files

Deploying

Development setup

Serving HTML

Serving static files

Optimizing autoreload

Why this design?

Caching

Code splitting

Putting it all together

One more thing!

Footnotes

Written by Aymeric Augustin