Django development workflow

Speak to 10 different Django developers and you will find 10 different ways of working. Speak to them the following year and you will no doubt find another 10 evolutions. This is just one of them, pickled in the moment.  I have a longish history with python but my Django experience is much more recent, but nonetheless I think this distills a some sound practices that I've learned from others, and I hope might be of help to others just starting out in Django.

Tools of the trade

OSX, Macports, Ubuntu Server, Vmware Fusion, Komodo IDE, Postgresql for my database, S3Hub, and Expandrive. Aesthetically I prefer textmate but code completion with tooltips, text completion, and mercurial integration are why I parted with more money and purchased Komodo IDE. Komodo is a little buggy at times but stability has improved since 5.1

Services

Amazon S3 for images, Ubuntu VSlices from MDhosting for (Australian) vm hosting, Bitbucket for repositories, Pingdom for website monitoring, and Canonical Landscape for vm monitoring. Landscape is more my way of giving back to Canonical for Ubuntu, because as a service in itself it is probably overpriced at this feature point.

Core Python packages

Django - tracking trunk of course. Virtualenv + wrapper, Pip, Mercurial for version control, and Fabric for deployment. The macport of Virtualenv + wrapper is recent but there is an issue with it adding 26 to the end of the link. Just need to rename the opt links. I Did work with Git for a while but find I get myself in less trouble with Mercurial especially when I need to revert. Maybe that's just me..

Django modules

South for migrations and Debug-toolbar are the two I can't live without. I would be curious if any people have others they absolutely can't go without installing in every project.

Project Structure

[domainname.com] > apacheconfs
                               > media
                               > logs
                               > wsgi_script
                               fabfile.py
                               > [projectname] > .hg
                                                         > apps > [internal_app_name]..
                                                         > project[s] > bin
                                                                           > templates
                                                                           manage.py
                                                                           > settings > __init__.py
                                                                                           (global settings)
                                                                                            dev_settings.py
                                                                                            staging_settings.py
                                                                                            prod_settings.py

                              

Daily Workflow

With a virtual env for each project with the projectname I create a postactivate script to cd into the manage.py directory with 'workon [projectname]'. Instead of running setup.py I usually symlink (I use a pylink script but I believe the virtualenv wrapper also does this) all external and internal apps into my virtual pythonpath unless for some reason it needs to explicitly be installed. I only put my own apps under my apps directory and even then I treat them as individual packages to be linked in - just to re-inforce in my head the concept of modular building blocks rather than one monolithic python package.

My internal Ubuntu Server staging vm is almost identical to production with only a few practical differences - the domain name and memory used by postgres, memcached etc. I pretty much follow the almanac for ubuntu setup of prod and staging. With Vmware fusion I fix the local vm's internal ip address, and alter the osx host file to point the staging domain to the vm.

At the project level, a Fabric fabfile defines setup() with all the tasks to setup the project on the server including creating the virtual env, cloning all the required repo's, and copying local files up to the server. The Fabric documentation examples don't appear to all work so I just define a setup_prod(), and setup_staging() which are decorated with the respective hosts and call the main setup(). My script looks something like this. I look forward to Fabric reaching 1.0, a tutorial being added to the docs, and the api stabilizing.

I use Mercurial hg update to switch between the default (trunk) branch and a dev branch. Most changes get made to the dev branch or if it's major surgery I'll clone the rep and create a new virtualenv. Fab deploy_staging pushes the dev branch changes to the staging server before I merge with default. Since the staging vm is for all intents and purposes identical to prod but with debug still switched on, there should be no nasty surprises when I merge with default and fab deploy_prod. The deploy script touches the djangowsgi.py script to effect the changes. There are additional fab functions for restarting apache, postgres, and updating any external packages as required.

Since the media files use version numbers in the filename I don't put media under version control and just use the nicely evolving S3Hub to organize my S3 media bucket, though I aim to include it in my fab script soon.

Development methodology

In terms of actual development I tend to work differently depending on the phase of the project. At the start I try and draw up the ultimate feature wishlist/architecture and then tear it down to the absolute minimum that is required to get the project into production and into alpha. In the same I break down my apps into simple modular components. I have seen too many monolithic projects fail because they spend too much time adding in requirements when really they should be removing. Pre 1.0 I cherry pick what tests I write. Tests become more important the closer you get to version 1.0 and start wanting to add all the other cool stuff in your wishlist. This is where you lose focus of the project as a whole and need the tests to make sure that while you're focusing on cool feature x you don't set fire to your house. I have whole projects I just can't bear revisiting because the testing foundations are pathetic. This is pretty much just agile development except I'm a little more pragmatic on test driven development pre 1.0.

References

There are a lot of good blog posts out there, but aside from the aforementioned Ubuntu Almanac I practically learnt Django and some sound 'its about the app stupid' practices from James Bennett's Practical Django Projects which is now infinitely more practical in it's 2nd edition. I'm also a big fan of Eric Holschers attempt to formalize conventions and evangelize testing. For some real project structures pinax and satchmo are nicely evolving projects which have already thought about this. Finally I couldn't live without the incredible effort that has gone into the main Django docs and django itself, which is making development a pleasure.

Parallel Importation of books - fueling the fire

Authors and publishers are horrified that the Productivity Commission (PC) has reduced our industry to a cultural externality to be balanced against economic rationalism, and argue the industry will be decimated, but I can't help feel they have a rose tinted view of where the industry is, and where it's going.

1. Unlike 'protectionism' of other Australian industries, the PC rightly points out that on balance the global publishers benefit far more from the current rules than Australians.

2. There are three classes of booksellers who are experiencing growth in the current Australian market, Amazon, the larger independents (who compete on their selection and engagement with readers) and most noticeably the discount department stores (DDS's) BigW, Target  and Kmart who are currently doing extremely well (greater than 10% growth) thanks in the most part to their globally competitive pricing.

3. Traditionally, department discounters (DDS) have focused on the pulp fiction sub $10 book market which is where the invisible giants, the romance and anonymous writers live, but also on the first release bestselling books where they can undercut the rest of the market. Because of the pressure globally on booksellers and higher wholesale price to booksellers in the protected Australian market, the DDS's are taking the bookseller's traditional ranged market and ironically making publishers pay extra for the privilege in volume rebates.

4. The multinational publishers target returns on average of 10-15%. How this is returned to the parent is irrelevant to the parent, though certainly the local publisher can balance a mix of staffing, new author development and pricing of their 'protected' successful authors. Their RRP will be influenced by the increased costs of dependence on economically efficient DDS's

5. Multinational publishers use their pricing protection advantage to outbid smaller publishers on advances for promising authors. Smaller publishers have to be more agile on staff costs and be prepared to risk lower returns to compete.

6. Authors base their fears on the way rights are currently traded, but they evolved that way because of PIRs. Since there is no benefit of fixed price rights post PIR, it stands that other contracts may become favoured which could equally be of greater overall benefit to authors or rights holders.

7. Forget cherry picking prices - the arguments are essentially about the average Australian bookseller being able to sell the US and UK mass market 2nd release bestsellers at $10-$15.00, at which price point books are less of a discretionary purchase, and there is less real price difference across the market between booksellers and discounters.

8. The elephant in the room is the electronic edition. Removal of PIRs in advance of the ebook becoming mainstream is practically doing the industry a favour in forcing it to adjust to the impact of the ebook against printed books.

9. Ironically booksellers, still somewhat suspicious of the campaign for the removal of PIRs voted best marketing campaign the revival of the most famous cheap books ever - the Penguin Classic, which is selling thousands of copies of popular backlist authors at $10 each in real bookstores.

10. Still not convinced? Imagine books are cigarettes. Now imagine the government put a 30% tax on cigarettes (which is about what they have done). Why do they do that? Well because it is proven the higher cost of cigarettes reduces the number of cigarettes sold, and the overall number of smokers and increases government revenue. Coincidentally the price of a packet of cigarettes is now about the same as a popular book, so the approx 30% 'tax' on popular 2nd release books reduces the number of readers and books sold for the sake of increased revenue. Except that unlike the tax on cigarettes 60% of the 'tax' goes to outside Australia, while the remaining 40% gets split amongst the local publishers/printers/retailers and authors.

 I leave the final words to the original Penguin Classics which were "generally viewed with suspicion and uncertainty by traditional publishers. Hardback fiction sold at seven or eight shillings, and it was feared that the new cheap paperbacks might undermine this market." One famous author opined: "In my capacity as a reader I applaud the Penguin Books; in my capacity as a writer I pronounce them anathema. Hutchinsons are now bringing out a very similar edition, though only of their own books, and if other publishers follow suit, the result may be a flood of cheap reprints which will cripple the lending libraries and check the output of new novels."

The more things change the more they stay the same.