A hidden gem in Django 1.7: ManifestStaticFilesStorage

The biggest change in Django 1.7 was the built-in schema migration support which everyone is aware of, however 1.7 also shipped with lots of other great additions, ManifestStaticFilesStorage - the new static files storage backend was one of them.

Static file caching is everywhere

Before explaining what ManifestStaticFilesStorage is and how it works, this is the overview of why we need it at Kogan.com:

cache busting static

In order to deliver the content to our customers as fast as possible, we cache the downloaded static files by using max-age request headers. This allows our customers to download the content once and the subsequent requests to static files will be served from a cache. As shown on the diagram, if we were to use normal static file names like base.css, the content of the file would be cached in the CDN as well as on the browser and we would have a hard time trying to invalidate these caches. We cache-bust the content by appending a md5 hash of the content of the file to the file name. When we deploy a new base.css, {% static %} template tag will turn base.css into base.d1833e.css and the browser will then request a new file. {% static %} template tag is able to translate base.css into base.d1833e.css thanks to static files storage backend. This setting is named STATICFILES_STORAGE in Django.

Before ManifestStaticFilesStorage

Our Django app was previously configured to use CachedStaticFilesStorage which resulted in placing file mappings in the CACHES backend, for us it was Redis. Django adds these mappings during collectstatic when it gathers all statics and puts them in one place.

collectstatic

This solution has coupled static assets deployment with code deployment resulting in a number of issues:

  • Running collectstatic as part of code deployment --> slow deploys
  • Extra load on Redis
  • App servers were sometimes out of sync as we deploy them in batch. When we start the deployment, Redis would be updated with the new keys, the first batch of App servers would get the new code, but the other half still had old code.

Out of sync app servers

ManifestStaticFilesStorage to the rescue

ManifestStaticFilesStorage has helped us to decouple the static compilation stage from deployments by allowing Django to read static file mappings from staticfiles.json on a filesystem. staticfiles.json is an artifact file produced by collectstatic with ManifestStaticFilesStorage as a backend. We can now include this staticfiles.json into our code package and deploy it to a single app server without affecting the others.

New ManifestStaticFilesStorage

Where is staticfiles.json located?

By default staticfiles.json will reside in STATIC_ROOT which is the directory where all static files are collected in. We host all our static assets on an S3 bucket which means staticfiles.json by default would end up being synced to S3. However, we wanted it to live in the code directory so we could package it and ship it to each app server. As a result of this, ManifestStaticFilesStorage will look for staticfiles.json in STATIC_ROOT in order to read the mappings. We had to overwrite this behaviour, so we subclassed ManifestStaticFilesStorage:


from django.contrib.staticfiles.storage import ManifestStaticFilesStorage
from django.conf import settings

class KoganManifestStaticFilesStorage(ManifestStaticFilesStorage):

    def read_manifest(self):
        """
        Looks up staticfiles.json in Project directory
        """
        manifest_location = os.path.abspath(
            os.path.join(settings.PROJECT_ROOT, self.manifest_name)
        )
        try:
            with open(manifest_location) as manifest:
                return manifest.read().decode('utf-8')
        except IOError:
            return None

With the above change, Django static template tag will now read the mappings from staticfiles.json that resides in project root directory.

Thanks Django

Thanks to Django 1.7, we've not only gotten a better schema migration system but also improved our deployment process. And not to mention ManifestStaticFilesStorage addition was only 40-50 lines of code (as of the day this blog post was published).

Nam Ngo