Test-Driven Development – How Do I Start?

There seem to be a lot of developers who like the idea of Test-Driven Development (TDD) and can clearly see the benefit of having tests written for their code but can’t seem to get their head around the process. How do you start writing unit tests before writing the actual code?

Let’s start with an example. You want to write a method that takes a URL as an argument and have it tell you through a boolean return if it’s the correct domain or not. It seems simple enough. Just write your method, pass the URL through some regular expression and you’re done.

But you yourself already know which domains are allowed and which are not so before writing the actual code you can run some tests in your head. E.g. I only want www.bbc.co.uk and its sub-domains on http and https. Nothing else should be allowed. So https://beta.bbc.co.uk/iplayer should return TRUE and http://www.bbcbb.com should return FALSE etc.

The process behind TDD is that you first write a failing test. Then you write the actual code and adjust until the test passes.

So let’s write some tests for our domain checker. I’m using Python and Unittest here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import unittest
 
# this is what we're going to be testing
class Utils():
    def is_bbc(self, val):
        pass #placeholder
 
# this is the actual test
class TestUtils(unittest.TestCase):
    def setUp(self):
        self.u = Utils()
 
    def test_is_bbc(self):
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://www.bbc.com'))
        self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
 
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)
unittest.TextTestRunner(verbosity=2).run(suite)

I’ve created a an empty method where our domain checker is going to live but as you can see it doesn’t do anything. The tests should immediately make sense. We pass a bunch of domain variations and we know which ones should pass or fail. Naturally, running the test right now will fail:

$ python sample.py 
test_is_bbc (__main__.TestUtils) ... FAIL
 
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "sample.py", line 14, in test_is_bbc
    self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
AssertionError
 
----------------------------------------------------------------------
Ran 1 test in 0.000s
 
FAILED (failures=1)

Now you can start writing the actual code and keep running the same tests until it passes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import unittest
import re
 
# this is what we're going to be testing
class Utils():
    def is_bbc(self, val):
        return re.match('^https?://([^/]+)?\.bbc\.co\.uk', val)
 
# this is the actual test
class TestUtils(unittest.TestCase):
    def setUp(self):
        self.u = Utils()
 
    def test_is_bbc(self):
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://www.bbc.com'))
        self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://.bbc.co.uk')) #this should fail
 
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)
unittest.TextTestRunner(verbosity=2).run(suite)

The regex may look like it’s correct but running the test will fail again:

$ python sample.py
test_is_bbc (__main__.TestUtils) ... FAIL
 
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "sample.py", line 22, in test_is_bbc
    self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
AssertionError
 
----------------------------------------------------------------------
Ran 1 test in 0.001s
 
FAILED (failures=1)

It has failed on the final assert because our code will also allow http://.bbc.co.uk and we obviously don’t want that. But as you can see we’ve caught this edge case before deploying our app so we can promptly fix our code.

Hopefully this example demonstrates why it’s a good idea to start with tests. This is obviously a simple example but on bigger projects predicting the outcome of your system can save you a lot of debugging time in the future.

Saving data with Titanium mobile

In my previous post I talked about getting a simple table view up and running quickly using Titanium. A few people have asked me to continue the tutorial and explain how to save individual items and then display saved items in a separate window.

Saving data is quite simple. You first need to create the database and schema. You obviously only need to do this the first time the user opens your app. So in your app.js file you can do something like this:

1
2

var db = Titanium.Database.open('mydb');
db.execute('CREATE TABLE IF NOT EXISTS SAVEDITEMS  (NAME TEXT)');

This should look very familiar to you if you’re used to SQL. It first instantiates a database object then checks to see if the “SAVEDITEMS” table exists. If not, it will simply create it. If it does exist, it will just ignore that command. This will create a table with a single text field for storing the name of a selected item.

Now that we have our table set up, we can read and write to it whenever we like. If you wanted to save an item, you would do this:

1

db.execute("INSERT INTO SAVEDITEMS ( NAME ) VALUES ('my cool item')");

To get a list of all items in the database:

1
2

var rows = db.execute("SELECT * FROM SAVEDITEMS");
rows.close();

The last line is very important. You must close your query results to avoid memory leakage.

You can now loop through the saved items and stick them in a table view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// create blank table view
var tableview = Titanium.UI.createTableView({});
Titanium.UI.currentWindow.add(tableview);
 
// create an empty array
var data = [];
 
// get a list of all saved items
var rows = db.execute('SELECT * FROM SAVEDITEMS');
 
// loop through all items
while (rows.isValidRow())
{
   // add each item to the data array and set the table row title    	
   data.push({title:rows.field(0),hasChild:true});
   rows.next();
}  
 
// load db data into the table view 
tableview.setData(data);
rows.close();

That should help you get started. You can look at how to use events to save items and also check to see which items have already been saved so you don’t show the save button etc. Take a look at the demo app I’ve created to see how I handled the saving logic. Specifically the details.js file.

Building a native mobile app with Titanium in less than an hour

When I wrote my last blog entry, I favoured objective-c over Appcelerator’s Titanium framework thinking it just wasn’t good enough. Almost 6 months later and I would like to take back that comment. Titanium is no longer an embedded web browser inside an app. It uses Javascript to compile an entire iPhone/Xcode project with objective-c, so it uses native UI elements using clever APIs. This is also true for Android apps, iPad and now Blackberry.

This is not to say there’s no need to use Xcode ever again. But if you’re planning to build a general navigation-based app with loads of menus, windows and the ability to pull/push data from the web, it can be done much faster with Titanium. Plus you’ll be able to create the same app for multiple platforms without re-writing the code.

So I wanted to put together a relatively simple app that lets me browse and search through a list of items, in this example, animals; and be able to group them alphabetically like the iPhone’s contacts directory. Then I want to click on an item to see more details and be able to save my favourite items for easy access later.

I’ve created a sample app that you can download and use right now. All you have to do is add your own data and you’re done.

I’m going to quickly run you through how to set up the tabs, the list, alphabetical grouping, search and detail view for each item. I won’t explain the saving feature here but once you do the main part below you should be able to easily follow the rest by looking at the code.

Here’s how it works. See the sample app to work out where to place individual files.

1. Create a tab group. One for the main list and one tab for saved items:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

// create tab group
var tabGroup = Titanium.UI.createTabGroup();
 
// create base UI tab and root window
var mainList = Titanium.UI.createWindow({  
    title:'Animals',
    backgroundColor:'#ffffff',
    barColor:'#333',
    url:'mainList.js',
    tabBarHidden: false
});
var tabMainList = Titanium.UI.createTab({  
    icon:'book.png',
    title:'Animals',
    window:mainList
});
 
// create saved items tab
var faves = Titanium.UI.createWindow({  
    title:'Saved Items',
    backgroundColor:'#ffffff',
    barColor:'#333',
    url:'faves.js',
    tabBarHidden: false
});
var tabFaves = Titanium.UI.createTab({  
    icon:'star.png',
    title:'Saved Items',
    window:faves
});
 
//  add tabs
tabGroup.addTab(tabMainList);
tabGroup.addTab(tabFaves);  
tabGroup.open(tabMainList);
 
Titanium.UI.currentWindow.add(tabGroup);

That’s it. We have now created two tabs. mainList.js will take care of the content for the first tab and faves.js for the saved items tab.

2. Create a JSON doc (data.js) containing your data for the list:

1
2
3
4
5
6
7

var myData = [
    {"title":"Armadillo", "description":"Armadillos eat ants but they are not Anteaters"},
    {"title":"Bear", "description":"Bears are awesome and they like honey"},
    {"title":"Dog", "description":"Dogs are the best and they woof"},
    {"title":"Deer", "description":"D'oh! a deer..."},
    {"title":"Zebra", "description":"Zebras are stripey and cool"}
];

3. Set up your directory list in the first tab (mainList.js)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

// include the data
Titanium.include('data.js');
 
var data = [];
var firstLetter;
var oldFirstLetter;
 
// loop through data and add to list
for (var i = 0; i < myData.length; i++)
{
	// get first letter of each item for grouping
	firstLetter = myData[i].title.substr(0,1);
	if(i == 0){
	    oldFirstLetter = firstLetter;
	    Ti.API.info(oldFirstLetter);
	} else {	
	    if(firstLetter == oldFirstLetter){
    	        oldFirstLetter = null;
    	    } else {
    	        oldFirstLetter = firstLetter;
    	    }
        }
        data.push({title:myData[i].title,hasChild:true,header:oldFirstLetter});	
        oldFirstLetter = firstLetter;
}
 
// add search bar
var search = Titanium.UI.createSearchBar({
	showCancel:false
});
 
// create table view
var tableview = Titanium.UI.createTableView({
    data:data,
    search:search,
    filterAttribute:'title'
});
 
Titanium.UI.currentWindow.add(tableview);
 
// create table view event listener
tableview.addEventListener('click', function(e)
{
    var win = Titanium.UI.createWindow({
		url:'details.js',
		backgroundColor:'#ffffff',
                barColor:'#333333',
		title:e.rowData.title
	});
 
	// send selected item's title to detail page
	win.currentItem = e.rowData.title;
 
	Titanium.UI.currentTab.open(win,{animated:true});
});

Notice in the tableview event listener we are telling it to just open a new window on click and pass the item’s title.

4. Display the selected item’s data in details.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

// include the data
Titanium.include('data.js');
 
win = Titanium.UI.currentWindow;
 
var desc = '';
 
// create scroll view for the content
var scrollView = Titanium.UI.createScrollView({
	contentWidth:'auto',
	contentHeight:'auto',
	top:0,
	showVerticalScrollIndicator:true,
	showHorizontalScrollIndicator:false
});
 
 
// loop through data and get correct item by title (this is easier than using Id's incase you add new items later)
for (var i = 0; i < myData.length; i++){
    if(myData[i].title == win.currentItem){
        desc = myData[i].description;
    }
}
 
// create description label
var label = Titanium.UI.createLabel({
	text:desc,
	height:'auto',
	width:300,
	top:10,
	font:{fontSize:16},
	color:'#333333',
	textAlign:'left'
});

That’s it. I said an hour to develop this but in reality it will probably take you much less once you get used to the APIs. Of course for the data source you can use XHR to get data from the web but if you want your app to be usable without an internet connection, you would have to store the data locally like this.

I hope this was helpful. Let me know if you have any questions.

Django Gravatar filter

If you want to add user avatars to your Django app, you can certainly use the excellent django-avatar app. This will let your users upload/edit their own avatars or use Gravatar.

But for my app I only wanted to use Gravatar so I was looking for a simpler solution that let me just pass the user object and an optional size in a template filter and have Gravatar take care of the rest.

The solution is custom template tags. If you’re already used to using the built-in template filters, you’ll know how useful and easy they are. I wanted my Gravatar filter to be as simple as possible. Something like this:

1

{{ user|gravatar:20 }}

Where 20 is the optional width/height of the avatar. This would then create an img tag with the full Gravatar URL.

First create your ‘templatetags’ directory and associated files as instructed in the docs. Then create a function that takes in the user object and uses the email address to construct the Gravatar URL:

1
2
3
4
5
6
7
8
9
10

from django import template
import hashlib
from django.utils.safestring import mark_safe
register = template.Library()
 
@register.filter()
def gravatar(user, size=50):
    gravatar_url = "http://www.gravatar.com/avatar"
    emailHash = hashlib.md5(user.email.lower()).hexdigest()
    return mark_safe("<img src='%s/%s.jpg?d=identicon&s=%s' alt='' />" % (gravatar_url, emailHash, size))

Django Geolocation

One thing I love about Django models is the ability to subclass its methods to add extra functionality without having to write any extra code in admin or view layers.

I have an application where a user can enter an address in the admin section. I want to plot this location on Google Maps later but I don’t want to have to parse the address and do a reverse geolocation lookup in the view layer every time that page is viewed. The best thing to do is to store the lat/long values in the database.

I could do this by messing around with the Django admin templates but I’d rather not even let the user know the geolocation lookup is happening. Besides, what if I want to interact with the DB from the interpreter? The geolocation bit should happen no matter where the database is being used.

This is my model

1
2
3
4
5
6
7
8
9

class Entry(models.Model):
    title = models.CharField(max_length=200)
    description = models.TextField('description', blank=True, null=True)
    address = models.CharField(max_length=200, blank=True, null=True)
    postcode = models.CharField(max_length=200, blank=True, null=True)
    city = models.CharField(max_length=200, blank=True, null=True)
    country = models.CharField(max_length=100)
    geo_lat = models.DecimalField('latitude', max_digits=13, decimal_places=10, blank=True, null=True)
    geo_long = models.DecimalField('longitude', max_digits=13, decimal_places=10, blank=True, null=True)

In Django admin I don’t show ‘geo_lat’ and ‘geo_lat’. We just ask the user to enter the address, then before saving, we do the lookup, set the lat/long values and then save the model.

Creating a new entry would still be done the same way from either admin, view or interpreter:

1
2

e = Entry(title='a new entry', address='123 Smith Road', city='London', country='UK')
e.save()

But we are going to hijack the save() method to do some extra work before saving to the database. Let’s create a method that does the geolocation lookup first:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import urllib
import urllib2
from django.utils import simplejson as json
 
def get_geo(address):
    address = urllib.quote(address)
    url = "http://maps.google.com/maps/geo?q=%s&output=json&oe=utf8&sensor=true_or_false&key=12345" % (address)
    data = urllib2.urlopen(url)
    obj = json.loads( data.read() )
    if obj['Status']['code'] == 200:
        data = obj['Placemark'][0]['Point']['coordinates']
    else:
        raise Exception('Invalid address')
    return data

We pass the address to Google and if we get a 200 status code, we grab the lat/long values and return them.

Now let’s call this method in our save() subclass (inside the Entry model):

1
2
3
4
5
6

def save(self):
    add = "%s, %s, %s, %s" % (self.address, self.postcode, self.city, self.country)
    geo_data = utils.get_geo(add)
    self.geo_long = str(geo_data[0])
    self.geo_lat = str(geo_data[1])
    super(Listing, self).save() # Call the "real" save() method

This could be improved so instead of throwing and exception for bad addresses we handle it more gracefully by informing the user or at least save the address and ignore the geolocation lookup. But either way, the model is now responsible for doing the extra work before saving the new/updated data.

App Engine Datastore API – Tutorial 2

In the last tutorial we quickly created a couple of models for storing our recipes and ingredients. In this tutorial we are going to assign ingredients to recipes as a one-to-many relationship.

But before we do that, we need to modify our data models a bit. The Recipe model stays the same but we’ll need a way of referencing the ingredients to add them to a single recipe. Like an SQL foreign key. We can do this using the “ReferenceProperty”:

1
2
3
4

class Ingredient(db.Model):
    name = db.StringProperty()
    recipe = db.ReferenceProperty(Recipe, collection_name='recipe_items')
    created = db.DateTimeProperty(auto_now_add=True)

The recipe ReferenceProperty allows us to associate Recipe objects to ingredients. Creating this relationship is straight forward:

1
2
3
4
5
6
7

# first create a recipe
recipe = Recipe(title = "Caprese Salad", description = "Light Italian classic").put()
 
# now create 3 ingredients and add them to the recipe object
Ingredient(recipe=recipe, name='Tomato').put()
Ingredient(recipe=recipe, name='Mozzarella').put()
Ingredient(recipe=recipe, name='Basil').put()

We just created 4 entries, 1 recipe and 3 ingredients. We can query them individually:

1
2

recipes = Recipe.all()
ingredients = Ingredient.all()

Except we’ve created a relationship between these entries so we can loop through all of our recipes and for each check to see if there are any associated ingredients. We can do this by using the collection_name we specified in our Ingredient model (“recipe_items”). So we can just get back a list of our recipes and send the whole list to the view like we did last time:

1
2
3
4
5
6

recipes = Recipe.all()
recipes.order("title")
recipeResults = recipes.fetch(limit=40)
 
# again, I'm just using Django templates
return render_to_response('main/index.html', {'recipeResults': recipeResults})

We don’t need to perform a separate query in the controller to get back the ingredients. All associated ingredient recipes are referenced so we can access them directly in the view:

1
2
3
4
5
6
7
8
9
10
11

<ul>
    {% for recipe in recipeResults %}
        <li>			
            {{ recipe.title }} 
#now for each recipe we'll loop through its associated ingredients
               ({% for ing in recipe.recipe_items %}
                   {{ ing.name }},
               {% endfor %})
        </li>
    {% endfor %}
</ul>

If you want to go further, I recommend checking out the modeling entity relationship docs. There are plenty of great examples.

App Engine Datastore API – Tutorial 1

I’ve spent the last few days messing about with Google App Engine and I have to admit I’m liking it – a lot!

As a web engineer, the most tedious part of developing for me is configuring environments and setting up databases. I just want to code and not have to worry about configuration and optimisation. So I guess that’s why I’m enjoying App Engine so much.

One of the really exciting aspects of developing in AE so far has been working with the modeling API. It’s nice not having to deal with messy SQL schemas and complicated joins. It’s all pure OO programming. There’s a lot of good documentation on the Google site but this will help you get going quickly.

For our sample app, we want to create a recipe site. We’ll keep it simple for now. We have recipes and each recipe has a bunch of ingredients so we’ll create an object that represents a single recipe and one that represent a single ingredient.

1
2
3
4
5
6
7
8

class Recipe(db.Model):
    title = db.StringProperty()
    description = db.TextProperty()
    created = db.DateTimeProperty(auto_now_add=True)
 
class Ingredient(db.Model):
    name = db.StringProperty()
    created = db.DateTimeProperty(auto_now_add=True)

This should be pretty straight forward. In SQL terms, we have created 2 tables with a bunch of fields and each field has a certain property like string, text, datetime etc. You don’t have to run any commands to create the database/tables. It’s all done at runtime.

Now that we have our models, let’s populate them.

1
2
3
4
5
6
7
8
9

# create a new recipe
recipe = Recipe()
 
# set the values of each field
recipe.title = "Caprese Salad"
recipe.description = "Light Italian classic"
 
# save it!
recipe.put

You can also do the above like this:

1

recipe = Recipe(title = "Caprese Salad", description = "Light Italian classic").put()

You can keep adding more recipes by creating a new instance of the Recipe model.

Now that we have a bunch of recipes, let’s get them all out and then output them in our view layer as html

1
2
3
4
5
6

recipes = Recipe.all()
recipes.order("title")
recipeResults = recipes.fetch(limit=40)
 
# I'm just using standard Django templates here
return render_to_response('index.html', {'recipeResults': recipeResults})

And in our template we can just loop through the results

1
2
3
4
5
6
7

<ul>
{% for recipe in recipeResults %}
	<li>
		{{ recipe.title }} - {{ recipe.description }} 
	</li>
{% endfor %}
</ul>

That’s it! and not a single line of SQL.

In the next tutorial we’ll create some ingredients and I’ll show how to do one-to-many relationships.

Memcached and Google App Engine

Memcached is your friend.

I have to admit I’m new to the caching world. In the past I’ve let my databases do the work for me. Caching mySQL queries and indexing helped a lot with speed and scalability and I never really knew how to optimise HTTP requests in my mashups (p.s. that’s the last time you’ll hear me use that horrid word).

When I started working at the BBC, I quickly realised my applications needed to scale better. We are no longer talking about a few hundred page loads an hour – more like millions. Thankfully, I learned a lot about scalability and the power of distributed memory caching. With memcached I was able to make fewer requests and this meant less load on the database and that made a drastic improvement on page load speeds.

Now I have a cheap shared host running my personal sites and it’s unlikely to have a service like memcache on these cheap plans but I was able to experiment on Google App Engine using Python and Google’s memcache service. I figured I would do a mini tutorial here as my first blog entry.

The function of memcache is really basic. Let’s say we want to grab my latest tweets and display them on my site. Let’s assume I get thousands of visitor on my site so every time someone loads my page, I have to make an HTTP request to Twitter’s API, get the data, parse it and display it.

This is unnecessary. All visitors would see the same list of tweets so why make a separate request for each? It makes more sense to cache the results the first time it is requested and then serve up the cached version to the rest of the visitors. That’s where memcache comes in. We can store the results of either an HTTP request or a database query for a set duration, say 60 seconds. By doing so, we’ve gone from making hundreds of HTTP requests a minute to just 1. This naturally improves performance and I’m sure Twitter’s operations guys will thank you for it.

Ok so let’s look at how we would do this in App Engine.

Let’s create a method called getTweets(). This will do one of two things. First it’ll ask memcache if there is an existing cache for my tweets. If so, it’ll return the list. Otherwise, make the HTTP call, get back data and store in memcache for the next user then return the data.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

from google.appengine.api import memcache
from django.utils import simplejson as json
 
data = getTweets()
 
def getTweets():
    data = memcache.get("twitter")
    if data is not None:
        return data
    else:
        result = urlfetch.fetch(url="http://twitter.com/statuses/user_timeline/givp.json")
        if result.status_code == 200:
            data = json.loads(result.content)
 
        memcache.add("twitter", data, 60)
        return data

That’s it! The thing using the getTweets() method doesn’t need to worry about how the data is being fetched because the returned results are exactly the same either way.

You’ll notice in the add and get methods I have used “twitter” as the first parameter. This is the key or identifier of the cached item and you can use whatever you like. For example, if you wanted to cache the tweets of multiple people, you could use a key like this: “twitter~person1″ and “twitter~person2″. As long as both the add and get methods use the same key.

And there you have it. My tweets are cached for 60 seconds. You can try this right now and actually see the difference. The first time you run this code it takes a while for the HTTP request but if you run it again after the initial execution the results are returned in a split second.

The numbers speak for themselves. These are average page load speeds:

Without memcache: 0.869s
With memcache: 0.148s