Test-Driven Development – How Do I Start?

There seem to be a lot of developers who like the idea of Test-Driven Development (TDD) and can clearly see the benefit of having tests written for their code but can’t seem to get their head around the process. How do you start writing unit tests before writing the actual code?

Let’s start with an example. You want to write a method that takes a URL as an argument and have it tell you through a boolean return if it’s the correct domain or not. It seems simple enough. Just write your method, pass the URL through some regular expression and you’re done.

But you yourself already know which domains are allowed and which are not so before writing the actual code you can run some tests in your head. E.g. I only want www.bbc.co.uk and its sub-domains on http and https. Nothing else should be allowed. So https://beta.bbc.co.uk/iplayer should return TRUE and http://www.bbcbb.com should return FALSE etc.

The process behind TDD is that you first write a failing test. Then you write the actual code and adjust until the test passes.

So let’s write some tests for our domain checker. I’m using Python and Unittest here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import unittest
 
# this is what we're going to be testing
class Utils():
    def is_bbc(self, val):
        pass #placeholder
 
# this is the actual test
class TestUtils(unittest.TestCase):
    def setUp(self):
        self.u = Utils()
 
    def test_is_bbc(self):
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://www.bbc.com'))
        self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
 
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)
unittest.TextTestRunner(verbosity=2).run(suite)

I’ve created a an empty method where our domain checker is going to live but as you can see it doesn’t do anything. The tests should immediately make sense. We pass a bunch of domain variations and we know which ones should pass or fail. Naturally, running the test right now will fail:

$ python sample.py 
test_is_bbc (__main__.TestUtils) ... FAIL
 
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "sample.py", line 14, in test_is_bbc
    self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
AssertionError
 
----------------------------------------------------------------------
Ran 1 test in 0.000s
 
FAILED (failures=1)

Now you can start writing the actual code and keep running the same tests until it passes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

import unittest
import re
 
# this is what we're going to be testing
class Utils():
    def is_bbc(self, val):
        return re.match('^https?://([^/]+)?\.bbc\.co\.uk', val)
 
# this is the actual test
class TestUtils(unittest.TestCase):
    def setUp(self):
        self.u = Utils()
 
    def test_is_bbc(self):
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))
        self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))
        self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://www.bbc.com'))
        self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))
        self.assertFalse(self.u.is_bbc('http://.bbc.co.uk')) #this should fail
 
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)
unittest.TextTestRunner(verbosity=2).run(suite)

The regex may look like it’s correct but running the test will fail again:

$ python sample.py
test_is_bbc (__main__.TestUtils) ... FAIL
 
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "sample.py", line 22, in test_is_bbc
    self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
AssertionError
 
----------------------------------------------------------------------
Ran 1 test in 0.001s
 
FAILED (failures=1)

It has failed on the final assert because our code will also allow http://.bbc.co.uk and we obviously don’t want that. But as you can see we’ve caught this edge case before deploying our app so we can promptly fix our code.

Hopefully this example demonstrates why it’s a good idea to start with tests. This is obviously a simple example but on bigger projects predicting the outcome of your system can save you a lot of debugging time in the future.

Saving data with Titanium mobile

In my previous post I talked about getting a simple table view up and running quickly using Titanium. A few people have asked me to continue the tutorial and explain how to save individual items and then display saved items in a separate window.

Saving data is quite simple. You first need to create the database and schema. You obviously only need to do this the first time the user opens your app. So in your app.js file you can do something like this:

1
2

var db = Titanium.Database.open('mydb');
db.execute('CREATE TABLE IF NOT EXISTS SAVEDITEMS  (NAME TEXT)');

This should look very familiar to you if you’re used to SQL. It first instantiates a database object then checks to see if the “SAVEDITEMS” table exists. If not, it will simply create it. If it does exist, it will just ignore that command. This will create a table with a single text field for storing the name of a selected item.

Now that we have our table set up, we can read and write to it whenever we like. If you wanted to save an item, you would do this:

1

db.execute("INSERT INTO SAVEDITEMS ( NAME ) VALUES ('my cool item')");

To get a list of all items in the database:

1
2

var rows = db.execute("SELECT * FROM SAVEDITEMS");
rows.close();

The last line is very important. You must close your query results to avoid memory leakage.

You can now loop through the saved items and stick them in a table view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// create blank table view
var tableview = Titanium.UI.createTableView({});
Titanium.UI.currentWindow.add(tableview);
 
// create an empty array
var data = [];
 
// get a list of all saved items
var rows = db.execute('SELECT * FROM SAVEDITEMS');
 
// loop through all items
while (rows.isValidRow())
{
   // add each item to the data array and set the table row title    	
   data.push({title:rows.field(0),hasChild:true});
   rows.next();
}  
 
// load db data into the table view 
tableview.setData(data);
rows.close();

That should help you get started. You can look at how to use events to save items and also check to see which items have already been saved so you don’t show the save button etc. Take a look at the demo app I’ve created to see how I handled the saving logic. Specifically the details.js file.

Building a native mobile app with Titanium in less than an hour

When I wrote my last blog entry, I favoured objective-c over Appcelerator’s Titanium framework thinking it just wasn’t good enough. Almost 6 months later and I would like to take back that comment. Titanium is no longer an embedded web browser inside an app. It uses Javascript to compile an entire iPhone/Xcode project with objective-c, so it uses native UI elements using clever APIs. This is also true for Android apps, iPad and now Blackberry.

This is not to say there’s no need to use Xcode ever again. But if you’re planning to build a general navigation-based app with loads of menus, windows and the ability to pull/push data from the web, it can be done much faster with Titanium. Plus you’ll be able to create the same app for multiple platforms without re-writing the code.

So I wanted to put together a relatively simple app that lets me browse and search through a list of items, in this example, animals; and be able to group them alphabetically like the iPhone’s contacts directory. Then I want to click on an item to see more details and be able to save my favourite items for easy access later.

I’ve created a sample app that you can download and use right now. All you have to do is add your own data and you’re done.

I’m going to quickly run you through how to set up the tabs, the list, alphabetical grouping, search and detail view for each item. I won’t explain the saving feature here but once you do the main part below you should be able to easily follow the rest by looking at the code.

Here’s how it works. See the sample app to work out where to place individual files.

1. Create a tab group. One for the main list and one tab for saved items:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

// create tab group
var tabGroup = Titanium.UI.createTabGroup();
 
// create base UI tab and root window
var mainList = Titanium.UI.createWindow({  
    title:'Animals',
    backgroundColor:'#ffffff',
    barColor:'#333',
    url:'mainList.js',
    tabBarHidden: false
});
var tabMainList = Titanium.UI.createTab({  
    icon:'book.png',
    title:'Animals',
    window:mainList
});
 
// create saved items tab
var faves = Titanium.UI.createWindow({  
    title:'Saved Items',
    backgroundColor:'#ffffff',
    barColor:'#333',
    url:'faves.js',
    tabBarHidden: false
});
var tabFaves = Titanium.UI.createTab({  
    icon:'star.png',
    title:'Saved Items',
    window:faves
});
 
//  add tabs
tabGroup.addTab(tabMainList);
tabGroup.addTab(tabFaves);  
tabGroup.open(tabMainList);
 
Titanium.UI.currentWindow.add(tabGroup);

That’s it. We have now created two tabs. mainList.js will take care of the content for the first tab and faves.js for the saved items tab.

2. Create a JSON doc (data.js) containing your data for the list:

1
2
3
4
5
6
7

var myData = [
    {"title":"Armadillo", "description":"Armadillos eat ants but they are not Anteaters"},
    {"title":"Bear", "description":"Bears are awesome and they like honey"},
    {"title":"Dog", "description":"Dogs are the best and they woof"},
    {"title":"Deer", "description":"D'oh! a deer..."},
    {"title":"Zebra", "description":"Zebras are stripey and cool"}
];

3. Set up your directory list in the first tab (mainList.js)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

// include the data
Titanium.include('data.js');
 
var data = [];
var firstLetter;
var oldFirstLetter;
 
// loop through data and add to list
for (var i = 0; i < myData.length; i++)
{
	// get first letter of each item for grouping
	firstLetter = myData[i].title.substr(0,1);
	if(i == 0){
	    oldFirstLetter = firstLetter;
	    Ti.API.info(oldFirstLetter);
	} else {	
	    if(firstLetter == oldFirstLetter){
    	        oldFirstLetter = null;
    	    } else {
    	        oldFirstLetter = firstLetter;
    	    }
        }
        data.push({title:myData[i].title,hasChild:true,header:oldFirstLetter});	
        oldFirstLetter = firstLetter;
}
 
// add search bar
var search = Titanium.UI.createSearchBar({
	showCancel:false
});
 
// create table view
var tableview = Titanium.UI.createTableView({
    data:data,
    search:search,
    filterAttribute:'title'
});
 
Titanium.UI.currentWindow.add(tableview);
 
// create table view event listener
tableview.addEventListener('click', function(e)
{
    var win = Titanium.UI.createWindow({
		url:'details.js',
		backgroundColor:'#ffffff',
                barColor:'#333333',
		title:e.rowData.title
	});
 
	// send selected item's title to detail page
	win.currentItem = e.rowData.title;
 
	Titanium.UI.currentTab.open(win,{animated:true});
});

Notice in the tableview event listener we are telling it to just open a new window on click and pass the item’s title.

4. Display the selected item’s data in details.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

// include the data
Titanium.include('data.js');
 
win = Titanium.UI.currentWindow;
 
var desc = '';
 
// create scroll view for the content
var scrollView = Titanium.UI.createScrollView({
	contentWidth:'auto',
	contentHeight:'auto',
	top:0,
	showVerticalScrollIndicator:true,
	showHorizontalScrollIndicator:false
});
 
 
// loop through data and get correct item by title (this is easier than using Id's incase you add new items later)
for (var i = 0; i < myData.length; i++){
    if(myData[i].title == win.currentItem){
        desc = myData[i].description;
    }
}
 
// create description label
var label = Titanium.UI.createLabel({
	text:desc,
	height:'auto',
	width:300,
	top:10,
	font:{fontSize:16},
	color:'#333333',
	textAlign:'left'
});

That’s it. I said an hour to develop this but in reality it will probably take you much less once you get used to the APIs. Of course for the data source you can use XHR to get data from the web but if you want your app to be usable without an internet connection, you would have to store the data locally like this.

I hope this was helpful. Let me know if you have any questions.

iPhone App Development with Google App Engine

After almost a year of messing around with various iPhone development alternatives such as Phonegap and Titanium, I finally decided to learn Objective-C and do it all properly. I actually think those other frameworks are brilliant as they allow you to use familiar languages like Javascript to quickly create nice apps for both iPhone and Android. But since they rely heavily on the web view element for loading HTML, creating sophisticated apps like Skype would be impossible.

So I set out to create an app for the iPhone with Objective-C. My app is pretty simple. It basically pulls in RSS news, audio podcast and video podcast feeds into a UITableView list, allowing the user to read, listen and watch news stories from the Democracy Now! website.

I managed to put together the app pretty quickly but I ran into a lot of issues when I tried to parse and massage the XML data. For starters, cocoa does not have native support for regular expressions (but there are several external libraries). I wanted to clean up the content I was getting back before displaying it to the user but I soon realised something that would normally take me a few minutes in Python/PHP/Javascript would take a lot longer in Objective-C. Parsing XML using NSXMLParser was an absolute nightmare and extremely slow. I rarely work with XML these days and find JSON a much easier protocol to deal with. I even tested the app with some sample JSON data using the excellent json-framework libarary and it was much easier and faster. Alas, I only had RSS feeds to work with.

The other problem I ran into was slow HTTP requests. It would sometimes take up to 20 seconds just to load the first screen. This was due to a combination of slow connection speeds, long response times from the data provider and a slow XML parser.

The solution I came up with was to do as little as possible in the phone app as far as the data was concerned. I decided to use Google App Engine to fetch the data from the source, parse, rejig, massage and beautify in Python, then serialise and return the results in JSON to the phone app to use.

It may sound like this would increase response times even more since the phone would have to first call GAE, then GAE would need to call the data source and then all the way back to the phone. This is true, however, once the data is with GAE we have the luxury of using memcache and datastore. The RSS and podcast feeds are updated once a day so there’s no reason to request the data from the source every time the user loads the app. Because each time we have to make the HTTP call, parse the data and load it up. This is extremely slow and unnecessary. We can just make one request a day, then parse, cleanup and cache the results for the next user that requests it.

So the app only talks to GAE. GAE first checks memcache to see if we have a cached version. If we don’t, it will make the HTTP call, fetch the data, parse, serialise, cache and return results. If we do have a cached version, there’s nothing else to do but to return the data. A cron job will also run every 24 hours to make sure memcache is up to date.

If you really want a solid and reliable app, you need to think about all the edge cases also. What happens if the cache expires and the data provider’s website is down? At that exact moment a user loads the app only to get an error message saying there’s nothing to show. An unlikely scenario but not impossible. So the way I got around this issue was to store the serialised JSON output in GAE’s datastore as well. We always use the data from memcache but should memcache be empty and the data source down, we can switch over to the datastore and load yesterday’s content instead. Not ideal but better than having a broken app.

This is a bit of an overkill for such a simple app but it’s super fast and efficient and will work well for almost any app that relies on 3rd-party APIs. To be fair, it was my lack of experience with Objective-C that led me to using GAE. I feel much more comfortable in Python than Objective-C and I’m sure an experienced cocoa developer would have no problems parsing and massaging data in the app itself.

Of course there is one other edge case – Google App Engine could go down or worst, the interwebz could break. In which case, a simple error message will suffice.

You can download Democracy Now! app on iTunes

KALX 90.7 FM iPhone App

My first iPhone/iPod Touch app is out! I’m waiting for the approval of a second app. When that’s done, I’ll do a proper post about all things Objective-C and iPhone SDK.

KALX app on iTunes

Soup

I’ve been meaning to get back into video editing for a while so I spent the weekend putting this together:

Soup from Giv Parvaneh on Vimeo.

Django Gravatar filter

If you want to add user avatars to your Django app, you can certainly use the excellent django-avatar app. This will let your users upload/edit their own avatars or use Gravatar.

But for my app I only wanted to use Gravatar so I was looking for a simpler solution that let me just pass the user object and an optional size in a template filter and have Gravatar take care of the rest.

The solution is custom template tags. If you’re already used to using the built-in template filters, you’ll know how useful and easy they are. I wanted my Gravatar filter to be as simple as possible. Something like this:

1

{{ user|gravatar:20 }}

Where 20 is the optional width/height of the avatar. This would then create an img tag with the full Gravatar URL.

First create your ‘templatetags’ directory and associated files as instructed in the docs. Then create a function that takes in the user object and uses the email address to construct the Gravatar URL:

1
2
3
4
5
6
7
8
9
10

from django import template
import hashlib
from django.utils.safestring import mark_safe
register = template.Library()
 
@register.filter()
def gravatar(user, size=50):
    gravatar_url = "http://www.gravatar.com/avatar"
    emailHash = hashlib.md5(user.email.lower()).hexdigest()
    return mark_safe("<img src='%s/%s.jpg?d=identicon&s=%s' alt='' />" % (gravatar_url, emailHash, size))

Test-driven Development in Agile Projects

I recently posted this on the BBC Web Developer Blog:

http://www.bbc.co.uk/blogs/webdeveloper/2009/11/testdriven-development-in-agil.shtml

Developers at the BBC tend to use Agile methodologies as a way to quickly release iterations of products. But where does rigorous code testing fit in with the short development and release cycles? How can we maintain the quality of our code when things need to change so fast?

More

Django Geolocation

One thing I love about Django models is the ability to subclass its methods to add extra functionality without having to write any extra code in admin or view layers.

I have an application where a user can enter an address in the admin section. I want to plot this location on Google Maps later but I don’t want to have to parse the address and do a reverse geolocation lookup in the view layer every time that page is viewed. The best thing to do is to store the lat/long values in the database.

I could do this by messing around with the Django admin templates but I’d rather not even let the user know the geolocation lookup is happening. Besides, what if I want to interact with the DB from the interpreter? The geolocation bit should happen no matter where the database is being used.

This is my model

1
2
3
4
5
6
7
8
9

class Entry(models.Model):
    title = models.CharField(max_length=200)
    description = models.TextField('description', blank=True, null=True)
    address = models.CharField(max_length=200, blank=True, null=True)
    postcode = models.CharField(max_length=200, blank=True, null=True)
    city = models.CharField(max_length=200, blank=True, null=True)
    country = models.CharField(max_length=100)
    geo_lat = models.DecimalField('latitude', max_digits=13, decimal_places=10, blank=True, null=True)
    geo_long = models.DecimalField('longitude', max_digits=13, decimal_places=10, blank=True, null=True)

In Django admin I don’t show ‘geo_lat’ and ‘geo_lat’. We just ask the user to enter the address, then before saving, we do the lookup, set the lat/long values and then save the model.

Creating a new entry would still be done the same way from either admin, view or interpreter:

1
2

e = Entry(title='a new entry', address='123 Smith Road', city='London', country='UK')
e.save()

But we are going to hijack the save() method to do some extra work before saving to the database. Let’s create a method that does the geolocation lookup first:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import urllib
import urllib2
from django.utils import simplejson as json
 
def get_geo(address):
    address = urllib.quote(address)
    url = "http://maps.google.com/maps/geo?q=%s&output=json&oe=utf8&sensor=true_or_false&key=12345" % (address)
    data = urllib2.urlopen(url)
    obj = json.loads( data.read() )
    if obj['Status']['code'] == 200:
        data = obj['Placemark'][0]['Point']['coordinates']
    else:
        raise Exception('Invalid address')
    return data

We pass the address to Google and if we get a 200 status code, we grab the lat/long values and return them.

Now let’s call this method in our save() subclass (inside the Entry model):

1
2
3
4
5
6

def save(self):
    add = "%s, %s, %s, %s" % (self.address, self.postcode, self.city, self.country)
    geo_data = utils.get_geo(add)
    self.geo_long = str(geo_data[0])
    self.geo_lat = str(geo_data[1])
    super(Listing, self).save() # Call the "real" save() method

This could be improved so instead of throwing and exception for bad addresses we handle it more gracefully by informing the user or at least save the address and ignore the geolocation lookup. But either way, the model is now responsible for doing the extra work before saving the new/updated data.

HTML Scraping – Python vs PHP (re-post)

I posted this last year on my old blog. The subject came up again today so I dug it up from the archives:

I usually hate these “X vs Y” discussions but this week I was working on a harvesting project and was trying to figure out whether I should go down the Python or PHP route. I have been using PHP for many years now so PHP was the obvious choice but recently I have been using Python a fair bit and the more I use it, the more I realise what a sloppy language PHP is.

So I set out to do some tests to see which would get my task done quicker.

For Python I used the Beautiful Soup module and for PHP I used PHP Simple HTML DOM Parser class.

The test:
Go to http://news.bbc.co.uk, look for the third <p> tag and return the text for the first link within that paragraph.

Both classes make this very easy to do so coding was not a concern:

in Python:

1
2
3

page = urllib2.urlopen("http://news.bbc.co.uk")
soup = BeautifulSoup(page)
print soup.findAll('p')[2].findAll('a')[0].string

in PHP:

1
2

$html = file_get_html('http://news.bbc.co.uk');
echo $html->find('p', 2)->find('a', 0)->innertext();

I ran each 5 times to measure the execution time.

Python:
0.622022151947 seconds
0.577415943146 seconds
0.518396139145 seconds
0.503247022629 seconds
0.482849121094 seconds

PHP:
0.430239915848 seconds
0.415632009506 seconds
0.408473014832 seconds
0.413187026978 seconds
0.411664962769 seconds

Pretty damn close but PHP is on average a bit faster it seems.

To be fair, this really isn’t a very good way to measure the performance of the two. The real test, in my opinion, would be to see how scalable each method is and how they handle memory management once I start scraping say the entire Wikipedia collection. I could be wrong here but from what I’ve read so far, Python is the tool of choice for such heavy processing tasks.

Either way, I think I’ll go with Python