Let’s start with an example. You want to write a method that takes a URL as an argument and have it tell you through a boolean return if it’s the correct domain or not. It seems simple enough. Just write your method, pass the URL through some regular expression and you’re done.

But you yourself already know which domains are allowed and which are not so before writing the actual code you can run some tests in your head. E.g. I only want www.bbc.co.uk and its sub-domains on http and https. Nothing else should be allowed. So https://beta.bbc.co.uk/iplayer should return TRUE and http://www.bbcbb.com should return FALSE etc.

The process behind TDD is that you first write a failing test. Then you write the actual code and adjust until the test passes.

So let’s write some tests for our domain checker. I’m using Python and Unittest here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import unittest
 
# this is what we're going to be testing
class Utils():
    def is_bbc(self, val):
        pass #placeholder
 
# this is the actual test
class TestUtils(unittest.TestCase):
    def setUp(self):
        self.u = Utils()
 
    def test_is_bbc(self):
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://www.bbc.com'))
        self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
 
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)
unittest.TextTestRunner(verbosity=2).run(suite)

I’ve created a an empty method where our domain checker is going to live but as you can see it doesn’t do anything. The tests should immediately make sense. We pass a bunch of domain variations and we know which ones should pass or fail. Naturally, running the test right now will fail:

$ python sample.py
test_is_bbc (__main__.TestUtils) ... FAIL
 
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "sample.py", line 14, in test_is_bbc
    self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
AssertionError
 
----------------------------------------------------------------------
Ran 1 test in 0.000s
 
FAILED (failures=1)

Now you can start writing the actual code and keep running the same tests until it passes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import unittest
import re
 
# this is what we're going to be testing
class Utils():
    def is_bbc(self, val):
        return re.match('^https?://([^/]+)?\.bbc\.co\.uk', val)
 
# this is the actual test
class TestUtils(unittest.TestCase):
    def setUp(self):
        self.u = Utils()
 
    def test_is_bbc(self):
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://www.bbc.com'))
        self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://.bbc.co.uk')) #this should fail
 
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)
unittest.TextTestRunner(verbosity=2).run(suite)

The regex may look like it’s correct but running the test will fail again:

$ python sample.py
test_is_bbc (__main__.TestUtils) ... FAIL
 
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "sample.py", line 22, in test_is_bbc
    self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
AssertionError
 
----------------------------------------------------------------------
Ran 1 test in 0.001s
 
FAILED (failures=1)

It has failed on the final assert because our code will also allow http://.bbc.co.uk and we obviously don’t want that. But as you can see we’ve caught this edge case before deploying our app so we can promptly fix our code.

Hopefully this example demonstrates why it’s a good idea to start with tests. This is obviously a simple example but on bigger projects predicting the outcome of your system can save you a lot of debugging time in the future.

So I set out to create an app for the iPhone with Objective-C. My app is pretty simple. It basically pulls in RSS news, audio podcast and video podcast feeds into a UITableView list, allowing the user to read, listen and watch news stories from the Democracy Now! website.

I managed to put together the app pretty quickly but I ran into a lot of issues when I tried to parse and massage the XML data. For starters, cocoa does not have native support for regular expressions (but there are several external libraries). I wanted to clean up the content I was getting back before displaying it to the user but I soon realised something that would normally take me a few minutes in Python/PHP/Javascript would take a lot longer in Objective-C. Parsing XML using NSXMLParser was an absolute nightmare and extremely slow. I rarely work with XML these days and find JSON a much easier protocol to deal with. I even tested the app with some sample JSON data using the excellent json-framework libarary and it was much easier and faster. Alas, I only had RSS feeds to work with.

The other problem I ran into was slow HTTP requests. It would sometimes take up to 20 seconds just to load the first screen. This was due to a combination of slow connection speeds, long response times from the data provider and a slow XML parser.

The solution I came up with was to do as little as possible in the phone app as far as the data was concerned. I decided to use Google App Engine to fetch the data from the source, parse, rejig, massage and beautify in Python, then serialise and return the results in JSON to the phone app to use.

It may sound like this would increase response times even more since the phone would have to first call GAE, then GAE would need to call the data source and then all the way back to the phone. This is true, however, once the data is with GAE we have the luxury of using memcache and datastore. The RSS and podcast feeds are updated once a day so there’s no reason to request the data from the source every time the user loads the app. Because each time we have to make the HTTP call, parse the data and load it up. This is extremely slow and unnecessary. We can just make one request a day, then parse, cleanup and cache the results for the next user that requests it.

So the app only talks to GAE. GAE first checks memcache to see if we have a cached version. If we don’t, it will make the HTTP call, fetch the data, parse, serialise, cache and return results. If we do have a cached version, there’s nothing else to do but to return the data. A cron job will also run every 24 hours to make sure memcache is up to date.

If you really want a solid and reliable app, you need to think about all the edge cases also. What happens if the cache expires and the data provider’s website is down? At that exact moment a user loads the app only to get an error message saying there’s nothing to show. An unlikely scenario but not impossible. So the way I got around this issue was to store the serialised JSON output in GAE’s datastore as well. We always use the data from memcache but should memcache be empty and the data source down, we can switch over to the datastore and load yesterday’s content instead. Not ideal but better than having a broken app.

This is a bit of an overkill for such a simple app but it’s super fast and efficient and will work well for almost any app that relies on 3rd-party APIs. To be fair, it was my lack of experience with Objective-C that led me to using GAE. I feel much more comfortable in Python than Objective-C and I’m sure an experienced cocoa developer would have no problems parsing and massaging data in the app itself.

Of course there is one other edge case – Google App Engine could go down or worst, the interwebz could break. In which case, a simple error message will suffice.

You can download Democracy Now! app on iTunes

But for my app I only wanted to use Gravatar so I was looking for a simpler solution that let me just pass the user object and an optional size in a template filter and have Gravatar take care of the rest.

The solution is custom template tags. If you’re already used to using the built-in template filters, you’ll know how useful and easy they are. I wanted my Gravatar filter to be as simple as possible. Something like this:

1

{{ user|gravatar:20 }}

Where 20 is the optional width/height of the avatar. This would then create an img tag with the full Gravatar URL.

First create your ‘templatetags’ directory and associated files as instructed in the docs. Then create a function that takes in the user object and uses the email address to construct the Gravatar URL:

1
2
3
4
5
6
7
8
9
10

from django import template
import hashlib
from django.utils.safestring import mark_safe
register = template.Library()
 
@register.filter()
def gravatar(user, size=50):
    gravatar_url = "http://www.gravatar.com/avatar"
    emailHash = hashlib.md5(user.email.lower()).hexdigest()
    return mark_safe("<img src='%s/%s.jpg?d=identicon&s=%s' alt='' />" % (gravatar_url, emailHash, size))

http://www.bbc.co.uk/blogs/webdeveloper/2009/11/testdriven-development-in-agil.shtml

Developers at the BBC tend to use Agile methodologies as a way to quickly release iterations of products. But where does rigorous code testing fit in with the short development and release cycles? How can we maintain the quality of our code when things need to change so fast?

More

I have an application where a user can enter an address in the admin section. I want to plot this location on Google Maps later but I don’t want to have to parse the address and do a reverse geolocation lookup in the view layer every time that page is viewed. The best thing to do is to store the lat/long values in the database.

I could do this by messing around with the Django admin templates but I’d rather not even let the user know the geolocation lookup is happening. Besides, what if I want to interact with the DB from the interpreter? The geolocation bit should happen no matter where the database is being used.

This is my model

1
2
3
4
5
6
7
8
9

class Entry(models.Model):
    title = models.CharField(max_length=200)
    description = models.TextField('description', blank=True, null=True)
    address = models.CharField(max_length=200, blank=True, null=True)
    postcode = models.CharField(max_length=200, blank=True, null=True)
    city = models.CharField(max_length=200, blank=True, null=True)
    country = models.CharField(max_length=100)
    geo_lat = models.DecimalField('latitude', max_digits=13, decimal_places=10, blank=True, null=True)
    geo_long = models.DecimalField('longitude', max_digits=13, decimal_places=10, blank=True, null=True)

In Django admin I don’t show ‘geo_lat’ and ‘geo_lat’. We just ask the user to enter the address, then before saving, we do the lookup, set the lat/long values and then save the model.

Creating a new entry would still be done the same way from either admin, view or interpreter:

1
2

e = Entry(title='a new entry', address='123 Smith Road', city='London', country='UK')
e.save()

But we are going to hijack the save() method to do some extra work before saving to the database. Let’s create a method that does the geolocation lookup first:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import urllib
import urllib2
from django.utils import simplejson as json
 
def get_geo(address):
    address = urllib.quote(address)
    url = "http://maps.google.com/maps/geo?q=%s&output=json&oe=utf8&sensor=true_or_false&key=12345" % (address)
    data = urllib2.urlopen(url)
    obj = json.loads( data.read() )
    if obj['Status']['code'] == 200:
        data = obj['Placemark'][0]['Point']['coordinates']
    else:
        raise Exception('Invalid address')
    return data

We pass the address to Google and if we get a 200 status code, we grab the lat/long values and return them.

Now let’s call this method in our save() subclass (inside the Entry model):

1
2
3
4
5
6

def save(self):
    add = "%s, %s, %s, %s" % (self.address, self.postcode, self.city, self.country)
    geo_data = utils.get_geo(add)
    self.geo_long = str(geo_data[0])
    self.geo_lat = str(geo_data[1])
    super(Listing, self).save() # Call the "real" save() method

This could be improved so instead of throwing and exception for bad addresses we handle it more gracefully by informing the user or at least save the address and ignore the geolocation lookup. But either way, the model is now responsible for doing the extra work before saving the new/updated data.