Fresh Tech: The Trek to Django 1.5

Fresh Tech: The Trek to Django 1.5

January 6, 2013

Last week, the Django project announced that the 1.5 release candidate was available for testing. For those of you for whom that last sentence was gibberish, Django is the software powering the Fresh Comics empire, and the 1.5 update is the new version of the software I've been awaiting to add some features to both the website and the apps to improve and expand the Fresh Comics experience.

While the main attraction of Django 1.5 is the customizable user model, I've also been using this as an excuse to upgrade a lot of Fresh Comics infrastructure to streamline work moving forward. The largest change thus far has been the switch from a MySQL database with locations provided by SOLR to a full GeoDjango stack built on the PostGIS extensions to the Postgres database. In the old system, I was basically limited to asking for things like store locations using queries like

Fetch me all stores located within 10 miles of location X, Y.

If I wanted to find the 20 nearest shops to a user's location, I would have to repeat this query multiple times if there my original query didn't return enough queries. In the code, this turns into the following procedure:

Let stores_list be a new empty list.

Let store_range be 5 miles.

While there are fewer than 20 stores in store_list:

Fetch me all stores located within store_range of location X, Y

Put the newfound stores in store_list

Double the value of store_range

In areas like Chicago, this loop would only run once or twice. In areas with fewer comic shops – like South Africa – this query would run many times until we found a range that would return enough stores. This creates a ton of work for the server, which translates into a slower service for everyone.

In the new GeoDjango-powered world, if I want to get the closest stores to a user, I pass the system a query like

Fetch the 20 closest stores to location X, Y and sort them in order of their distance from the user.

Multiple queries become a single query and things run much faster.

In addition to improving the existing system, having a full geospatial system at my command also allows me to do things not possible under the old system. One of the things that I'm most excited about is applied cluster analysis. The first new problem that I'm looking forward to tackling is correlating user activity with local information – in other words, focusing my limited resources on improving service for areas where Fresh Comics is most heavily used. For example, I can find the areas where user density is highest and test whether I have enough stores and events listed in the app in those areas. If I find a dense area of activity, but there are few stores or event listings, this becomes a signal for me focus some attention on that part of the world and either find shops in that area or start to work with online-only retailers to promote their services to users in those areas.

I've been testing out the software that I'll be using to transition the software on a test server and the Django folks have done a great job making the migration path from 1.3 to 1.5 a smooth experience. I have a full battery of tests that I'm using to highlight issues to address, but should the Django folks release the official 1.5 software this week, I should be ready to migrate everything next weekend.

Then things will get really interesting. Stay tuned.

comments powered by Disqus