HTML Scraping – Python vs PHP (re-post)

I posted this last year on my old blog. The subject came up again today so I dug it up from the archives:

I usually hate these “X vs Y” discussions but this week I was working on a harvesting project and was trying to figure out whether I should go down the Python or PHP route. I have been using PHP for many years now so PHP was the obvious choice but recently I have been using Python a fair bit and the more I use it, the more I realise what a sloppy language PHP is.

So I set out to do some tests to see which would get my task done quicker.

For Python I used the Beautiful Soup module and for PHP I used PHP Simple HTML DOM Parser class.

The test:
Go to http://news.bbc.co.uk, look for the third <p> tag and return the text for the first link within that paragraph.

Both classes make this very easy to do so coding was not a concern:

in Python:

1
2
3

page = urllib2.urlopen("http://news.bbc.co.uk")
soup = BeautifulSoup(page)
print soup.findAll('p')[2].findAll('a')[0].string

in PHP:

1
2

$html = file_get_html('http://news.bbc.co.uk');
echo $html->find('p', 2)->find('a', 0)->innertext();

I ran each 5 times to measure the execution time.

Python:
0.622022151947 seconds
0.577415943146 seconds
0.518396139145 seconds
0.503247022629 seconds
0.482849121094 seconds

PHP:
0.430239915848 seconds
0.415632009506 seconds
0.408473014832 seconds
0.413187026978 seconds
0.411664962769 seconds

Pretty damn close but PHP is on average a bit faster it seems.

To be fair, this really isn’t a very good way to measure the performance of the two. The real test, in my opinion, would be to see how scalable each method is and how they handle memory management once I start scraping say the entire Wikipedia collection. I could be wrong here but from what I’ve read so far, Python is the tool of choice for such heavy processing tasks.

Either way, I think I’ll go with Python