Django development workflow

Speak to 10 different Django developers and you will find 10 different ways of working. Speak to them the following year and you will no doubt find another 10 evolutions. This is just one of them, pickled in the moment.  I have a longish history with python but my Django experience is much more recent, but nonetheless I think this distills a some sound practices that I've learned from others, and I hope might be of help to others just starting out in Django.

Tools of the trade

OSX, Macports, Ubuntu Server, Vmware Fusion, Komodo IDE, Postgresql for my database, S3Hub, and Expandrive. Aesthetically I prefer textmate but code completion with tooltips, text completion, and mercurial integration are why I parted with more money and purchased Komodo IDE. Komodo is a little buggy at times but stability has improved since 5.1

Services

Amazon S3 for images, Ubuntu VSlices from MDhosting for (Australian) vm hosting, Bitbucket for repositories, Pingdom for website monitoring, and Canonical Landscape for vm monitoring. Landscape is more my way of giving back to Canonical for Ubuntu, because as a service in itself it is probably overpriced at this feature point.

Core Python packages

Django - tracking trunk of course. Virtualenv + wrapper, Pip, Mercurial for version control, and Fabric for deployment. The macport of Virtualenv + wrapper is recent but there is an issue with it adding 26 to the end of the link. Just need to rename the opt links. I Did work with Git for a while but find I get myself in less trouble with Mercurial especially when I need to revert. Maybe that's just me..

Django modules

South for migrations and Debug-toolbar are the two I can't live without. I would be curious if any people have others they absolutely can't go without installing in every project.

Project Structure

[domainname.com] > apacheconfs
                               > media
                               > logs
                               > wsgi_script
                               fabfile.py
                               > [projectname] > .hg
                                                         > apps > [internal_app_name]..
                                                         > project[s] > bin
                                                                           > templates
                                                                           manage.py
                                                                           > settings > __init__.py
                                                                                           (global settings)
                                                                                            dev_settings.py
                                                                                            staging_settings.py
                                                                                            prod_settings.py

                              

Daily Workflow

With a virtual env for each project with the projectname I create a postactivate script to cd into the manage.py directory with 'workon [projectname]'. Instead of running setup.py I usually symlink (I use a pylink script but I believe the virtualenv wrapper also does this) all external and internal apps into my virtual pythonpath unless for some reason it needs to explicitly be installed. I only put my own apps under my apps directory and even then I treat them as individual packages to be linked in - just to re-inforce in my head the concept of modular building blocks rather than one monolithic python package.

My internal Ubuntu Server staging vm is almost identical to production with only a few practical differences - the domain name and memory used by postgres, memcached etc. I pretty much follow the almanac for ubuntu setup of prod and staging. With Vmware fusion I fix the local vm's internal ip address, and alter the osx host file to point the staging domain to the vm.

At the project level, a Fabric fabfile defines setup() with all the tasks to setup the project on the server including creating the virtual env, cloning all the required repo's, and copying local files up to the server. The Fabric documentation examples don't appear to all work so I just define a setup_prod(), and setup_staging() which are decorated with the respective hosts and call the main setup(). My script looks something like this. I look forward to Fabric reaching 1.0, a tutorial being added to the docs, and the api stabilizing.

I use Mercurial hg update to switch between the default (trunk) branch and a dev branch. Most changes get made to the dev branch or if it's major surgery I'll clone the rep and create a new virtualenv. Fab deploy_staging pushes the dev branch changes to the staging server before I merge with default. Since the staging vm is for all intents and purposes identical to prod but with debug still switched on, there should be no nasty surprises when I merge with default and fab deploy_prod. The deploy script touches the djangowsgi.py script to effect the changes. There are additional fab functions for restarting apache, postgres, and updating any external packages as required.

Since the media files use version numbers in the filename I don't put media under version control and just use the nicely evolving S3Hub to organize my S3 media bucket, though I aim to include it in my fab script soon.

Development methodology

In terms of actual development I tend to work differently depending on the phase of the project. At the start I try and draw up the ultimate feature wishlist/architecture and then tear it down to the absolute minimum that is required to get the project into production and into alpha. In the same I break down my apps into simple modular components. I have seen too many monolithic projects fail because they spend too much time adding in requirements when really they should be removing. Pre 1.0 I cherry pick what tests I write. Tests become more important the closer you get to version 1.0 and start wanting to add all the other cool stuff in your wishlist. This is where you lose focus of the project as a whole and need the tests to make sure that while you're focusing on cool feature x you don't set fire to your house. I have whole projects I just can't bear revisiting because the testing foundations are pathetic. This is pretty much just agile development except I'm a little more pragmatic on test driven development pre 1.0.

References

There are a lot of good blog posts out there, but aside from the aforementioned Ubuntu Almanac I practically learnt Django and some sound 'its about the app stupid' practices from James Bennett's Practical Django Projects which is now infinitely more practical in it's 2nd edition. I'm also a big fan of Eric Holschers attempt to formalize conventions and evangelize testing. For some real project structures pinax and satchmo are nicely evolving projects which have already thought about this. Finally I couldn't live without the incredible effort that has gone into the main Django docs and django itself, which is making development a pleasure.