Django Gravatar filter

If you want to add user avatars to your Django app, you can certainly use the excellent django-avatar app. This will let your users upload/edit their own avatars or use Gravatar.

But for my app I only wanted to use Gravatar so I was looking for a simpler solution that let me just pass the user object and an optional size in a template filter and have Gravatar take care of the rest.

The solution is custom template tags. If you’re already used to using the built-in template filters, you’ll know how useful and easy they are. I wanted my Gravatar filter to be as simple as possible. Something like this:

1

{{ user|gravatar:20 }}

Where 20 is the optional width/height of the avatar. This would then create an img tag with the full Gravatar URL.

First create your ‘templatetags’ directory and associated files as instructed in the docs. Then create a function that takes in the user object and uses the email address to construct the Gravatar URL:

1
2
3
4
5
6
7
8
9
10

from django import template
import hashlib
from django.utils.safestring import mark_safe
register = template.Library()
 
@register.filter()
def gravatar(user, size=50):
    gravatar_url = "http://www.gravatar.com/avatar"
    emailHash = hashlib.md5(user.email.lower()).hexdigest()
    return mark_safe("<img src='%s/%s.jpg?d=identicon&s=%s' alt='' />" % (gravatar_url, emailHash, size))

Test-driven Development in Agile Projects

I recently posted this on the BBC Web Developer Blog:

http://www.bbc.co.uk/blogs/webdeveloper/2009/11/testdriven-development-in-agil.shtml

Developers at the BBC tend to use Agile methodologies as a way to quickly release iterations of products. But where does rigorous code testing fit in with the short development and release cycles? How can we maintain the quality of our code when things need to change so fast?

More

Django Geolocation

One thing I love about Django models is the ability to subclass its methods to add extra functionality without having to write any extra code in admin or view layers.

I have an application where a user can enter an address in the admin section. I want to plot this location on Google Maps later but I don’t want to have to parse the address and do a reverse geolocation lookup in the view layer every time that page is viewed. The best thing to do is to store the lat/long values in the database.

I could do this by messing around with the Django admin templates but I’d rather not even let the user know the geolocation lookup is happening. Besides, what if I want to interact with the DB from the interpreter? The geolocation bit should happen no matter where the database is being used.

This is my model

1
2
3
4
5
6
7
8
9

class Entry(models.Model):
    title = models.CharField(max_length=200)
    description = models.TextField('description', blank=True, null=True)
    address = models.CharField(max_length=200, blank=True, null=True)
    postcode = models.CharField(max_length=200, blank=True, null=True)
    city = models.CharField(max_length=200, blank=True, null=True)
    country = models.CharField(max_length=100)
    geo_lat = models.DecimalField('latitude', max_digits=13, decimal_places=10, blank=True, null=True)
    geo_long = models.DecimalField('longitude', max_digits=13, decimal_places=10, blank=True, null=True)

In Django admin I don’t show ‘geo_lat’ and ‘geo_lat’. We just ask the user to enter the address, then before saving, we do the lookup, set the lat/long values and then save the model.

Creating a new entry would still be done the same way from either admin, view or interpreter:

1
2

e = Entry(title='a new entry', address='123 Smith Road', city='London', country='UK')
e.save()

But we are going to hijack the save() method to do some extra work before saving to the database. Let’s create a method that does the geolocation lookup first:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import urllib
import urllib2
from django.utils import simplejson as json
 
def get_geo(address):
    address = urllib.quote(address)
    url = "http://maps.google.com/maps/geo?q=%s&output=json&oe=utf8&sensor=true_or_false&key=12345" % (address)
    data = urllib2.urlopen(url)
    obj = json.loads( data.read() )
    if obj['Status']['code'] == 200:
        data = obj['Placemark'][0]['Point']['coordinates']
    else:
        raise Exception('Invalid address')
    return data

We pass the address to Google and if we get a 200 status code, we grab the lat/long values and return them.

Now let’s call this method in our save() subclass (inside the Entry model):

1
2
3
4
5
6

def save(self):
    add = "%s, %s, %s, %s" % (self.address, self.postcode, self.city, self.country)
    geo_data = utils.get_geo(add)
    self.geo_long = str(geo_data[0])
    self.geo_lat = str(geo_data[1])
    super(Listing, self).save() # Call the "real" save() method

This could be improved so instead of throwing and exception for bad addresses we handle it more gracefully by informing the user or at least save the address and ignore the geolocation lookup. But either way, the model is now responsible for doing the extra work before saving the new/updated data.

HTML Scraping – Python vs PHP (re-post)

I posted this last year on my old blog. The subject came up again today so I dug it up from the archives:

I usually hate these “X vs Y” discussions but this week I was working on a harvesting project and was trying to figure out whether I should go down the Python or PHP route. I have been using PHP for many years now so PHP was the obvious choice but recently I have been using Python a fair bit and the more I use it, the more I realise what a sloppy language PHP is.

So I set out to do some tests to see which would get my task done quicker.

For Python I used the Beautiful Soup module and for PHP I used PHP Simple HTML DOM Parser class.

The test:
Go to http://news.bbc.co.uk, look for the third <p> tag and return the text for the first link within that paragraph.

Both classes make this very easy to do so coding was not a concern:

in Python:

1
2
3

page = urllib2.urlopen("http://news.bbc.co.uk")
soup = BeautifulSoup(page)
print soup.findAll('p')[2].findAll('a')[0].string

in PHP:

1
2

$html = file_get_html('http://news.bbc.co.uk');
echo $html->find('p', 2)->find('a', 0)->innertext();

I ran each 5 times to measure the execution time.

Python:
0.622022151947 seconds
0.577415943146 seconds
0.518396139145 seconds
0.503247022629 seconds
0.482849121094 seconds

PHP:
0.430239915848 seconds
0.415632009506 seconds
0.408473014832 seconds
0.413187026978 seconds
0.411664962769 seconds

Pretty damn close but PHP is on average a bit faster it seems.

To be fair, this really isn’t a very good way to measure the performance of the two. The real test, in my opinion, would be to see how scalable each method is and how they handle memory management once I start scraping say the entire Wikipedia collection. I could be wrong here but from what I’ve read so far, Python is the tool of choice for such heavy processing tasks.

Either way, I think I’ll go with Python

App Engine Datastore API – Tutorial 2

In the last tutorial we quickly created a couple of models for storing our recipes and ingredients. In this tutorial we are going to assign ingredients to recipes as a one-to-many relationship.

But before we do that, we need to modify our data models a bit. The Recipe model stays the same but we’ll need a way of referencing the ingredients to add them to a single recipe. Like an SQL foreign key. We can do this using the “ReferenceProperty”:

1
2
3
4

class Ingredient(db.Model):
    name = db.StringProperty()
    recipe = db.ReferenceProperty(Recipe, collection_name='recipe_items')
    created = db.DateTimeProperty(auto_now_add=True)

The recipe ReferenceProperty allows us to associate Recipe objects to ingredients. Creating this relationship is straight forward:

1
2
3
4
5
6
7

# first create a recipe
recipe = Recipe(title = "Caprese Salad", description = "Light Italian classic").put()
 
# now create 3 ingredients and add them to the recipe object
Ingredient(recipe=recipe, name='Tomato').put()
Ingredient(recipe=recipe, name='Mozzarella').put()
Ingredient(recipe=recipe, name='Basil').put()

We just created 4 entries, 1 recipe and 3 ingredients. We can query them individually:

1
2

recipes = Recipe.all()
ingredients = Ingredient.all()

Except we’ve created a relationship between these entries so we can loop through all of our recipes and for each check to see if there are any associated ingredients. We can do this by using the collection_name we specified in our Ingredient model (“recipe_items”). So we can just get back a list of our recipes and send the whole list to the view like we did last time:

1
2
3
4
5
6

recipes = Recipe.all()
recipes.order("title")
recipeResults = recipes.fetch(limit=40)
 
# again, I'm just using Django templates
return render_to_response('main/index.html', {'recipeResults': recipeResults})

We don’t need to perform a separate query in the controller to get back the ingredients. All associated ingredient recipes are referenced so we can access them directly in the view:

1
2
3
4
5
6
7
8
9
10
11

<ul>
    {% for recipe in recipeResults %}
        <li>
            {{ recipe.title }}
#now for each recipe we'll loop through its associated ingredients
               ({% for ing in recipe.recipe_items %}
                   {{ ing.name }},
               {% endfor %})
        </li>
    {% endfor %}
</ul>

If you want to go further, I recommend checking out the modeling entity relationship docs. There are plenty of great examples.