Django and Google App Engine

Reading Jacob Kaplan-Moss's post Snakes on the Web my first thought was - why hasn't he mentioned Google App Engine? To be fair to Jacob, the GAE sdk when launched was pretty underwhelming but GAE is the ugly duckling that could very well grow into a swan. If Django is for 'perfectionists with deadlines' then GAE might be said to be for 'perfectionists with aspirations'.

The growth of Django compared with say Zope I think is in part due to it's dependence on relational databases and their ubiquity in business. Google is such a disruptive force that Bigtable could well become the next data foundation upon which enterprises are built in the same way DB2 and Oracle were for the relational era. Large enterprises are risk averse with data, but Google is a big enough vendor to threaten the status quo.

Much as Django tries to hide the relational model from the developer, the further we get down the development path with future features like multiple database support and recent additions such as aggregation the more it becomes apparent that we're exposing more problems in the ORM pattern than we're reducing.

Bigtable makes me think about the financials product we use (Microsoft Dynamics Nav - formerly Navision). Upgrading from the native Navision database to MS SQL Server required at least twice the hardware capacity and actually reduced performance. Why? well although the native database didn't scale, it had aspects that remind me of BigTable. It had an index called a Sum Index Flow Field (SIFT). This index was dimensioned by time and other optional dimensions.  It took ages to build the index, but filtering by date range and dimension a sum value could be computed with just two records at either end of the date range. Contrast this to the MS SQL relational approach which required an index on the primary key, dimension and value and needed to add up ALL the relevent records each time it was accessed.

While prototyping a Django 1.1 business intelligence app for internal use I was starting to realise why I want just want queues to be a standard feature and why I actually don't want to use aggregations no matter how tempting they seem. Stuffing data into memcache like a christmas turkey is not as smart as I thought it was either, and no matter how great South is, data migrations are just not exciting.

Now that it has Task Queues, and other parallel computing goodies it seems like there is even less reason not to try Google App Engine. Ironically the only thing that is holding me back is that I need to downgrade my Snow Leopard python interpreter to 2.5 after being delighted at seeing 2.6 installed as the default.

Snow Leopard + Python = :)

[update 14-Sep-09]
Macports + python26 has been patched and is working correctly. The only issue left which is easy to work around is virtualenv. I install pretty much everything in the virtual environments and just leave the tricky or base packages to macports so here are the select packages that get installed via macports, and which now install nicely.

port install python26 
rinse and repeat for python_select (then python_select python26), py26-pip,  py26-fabric, py26-pil, mercurial, git-core

Finally to workaround a virtualenv snow leopard bug in 1.3.3 easy_install-2.6 virtualenv==dev + don't forget virtualenvwrapper.

I prefer to use macports + python26 + virtualenv but the python26 port is broken at the moment. There are documented workarounds which gets it up and running but I was still missing the _locale module which in turn broke the mercurial port etc etc.

In the interim I thought I'd use the default x64 2.6.1 that comes installed, and was pleasantly surprised. For one, they've moved the site-packages to /Library/Python/2.6/site-packages which just seems a better place for it to live. Secondly to kick off installation of a few essential packages such as virtualenv I noticed easy_install is in /usr/bin this time around. That makes me one happy camper.

Beyond python Snow Leopard seems a very subtle upgrade for the user. It seems silly but the new Expose feature showing minimized windows, combined with the minimize to application preference was my number one reason for upgrading.  I enjoy the aesthetics of OSX more than other aspects of it, but minimize was one ui inconsistency which always irritated.

The other thing I really like is the fact there are no serial numbers to input and no forced earlier os requirements - just an intel mac. It's an interesting move which moves the OS to the backseat and makes it all about the hardware and apps. People might say it's only a service pack - which is superficially true, but modest changes for the user make it easier to keep people on the upgrade treadmill, especially when driven by new apps that take advantage of the hidden technologies of Snow Leopard. Microsoft faces an uphill battle to get people to upgrade at their price points. For those that have adopted and survived the initial horror of Vista, it's an expensive ask to upgrade to Windows 7, and for everyone else still on XP Microsoft pretty much expects it's resellers to sell it with a new computer. The only real hope for MS is they can limit the headway of osx now its looking more enterprise ready.