Traditional relational databases (mySQL PostgreSQL etc) and noSQL systems are not mutually exclusive. I have several Django applications that are happily using mySQL. If your site is not scaling due to your database, you are doing it wrong! noSQL will not help you until you start caching some of those expensive queries using something like Memcached.
I use MongoDB alongside mySQL for all the dirty work like storing stats for later processing. There’s no point in polluting mySQL with this sort of data, especially when you’re dealing with millions of entries.
This post is intended for absolute beginners who use Django tranditionally and are curious about how they can integrate a secondary storage service into their apps. I’m assuming you have already installed MongoDB on your dev environment. You will also need to install the MongoEngine library for Python.
Let’s start.
You already know how to create data models in Django, but let’s say we want to store an activity feed for your users everytime they do something on your site. We begin by creating a data model similar to Django’s ORM using MongoEngine but the difference here is that you don’t need to run “syncdb” to create your tables. Mongo’s collections (similar to SQL tables) are schemaless so these models can be manipulated and you won’t need to worry about running migration scripts.
Let’s create a simple collection for storing user activities. Create a file where you normally keep your Django models and call it mongomodel.py
1
2
3
4
5
6
7
8
9
10
11
12
from mongoengine import*# connect to a db (no need to create this - it will be created automagically)
connect('useractivity')class Author(Document):
pk = IntField()
name = StringField(max_length=200, required=True)class Activity(Document):
message = StringField(max_length=200)
author = ReferenceField(Author, reverse_delete_rule=CASCADE)
“What’s this??!! Django already has a User model, why do I need another in Mongo?” Well, you don’t, but say you want your activity to say something like: “Joe uploaded a photo” and you want Joe’s name to be linked to his profile page. We keep a reference to his mySQL id in case we need to look up other info or construct a URL.
You’ll also notice in the Activity model we are referencing the Author model. This is like a foreign key that will allow us to create relationships, similar to SQL. The CASCADE option will make sure if the user is deleted, all activities are also cleared out.
Ok, let’s start using this puppy! Using the example above we want to create an activity for Joe next time he uploads a photo. First, import mongomodel.py whenever you’re planning to interact with Mongo. In my photo upload view function I will create an activity like so:
1
2
3
4
5
6
7
8
9
10
11
# After photo upload is completefrom main.mongomodelimport*# first create a user object - you can grab data from request object
the_author = Author(pk=request.user.id, name=request.user.first_name)
the_author.save()# now create the activity
activity = Activity(message='uploaded a new photo', author=the_author)
activity.save()
That’s it. If you decide later you also want to add the name of the file uploaded you can simply add a new field to your Activity model and it will just work, plus it will be backwards compatible, i.e. older records without this field will not complain. Lovely.
Displaying the activity is just as simple. In your view function pull out the record and push down to your template:
1
2
3
4
5
6
7
from main.mongomodelimport*# get all activities
activities = Activity.objects# push down to templatereturn render_to_response('activities.html', {'activities':activites})
Now in your template loop and output like any other model:
1
2
3
4
5
<ul>{%for a in activites %}<li><a href="{% url main.views.profile a.author.pk %}">{{ a.author.name}}</a>{{ a.message}}</li>{% endfor %}</ul>
I’ve used the user’s mySQL primary key to construct his profile URL.
This is a very basic example but hopefully you can see the advantage of offloading some of the data storage to Mongo. You may ask “but what if the user changes his name? won’t the data in the activity remain out of sync?”. Yes, it will, but you can very easily add a simple method in your Django user model to update Mongo records whenever the user’s details are updated.
This is not another SQL vs noSQL rant. I’m not here to defend one or the other and there are plenty of articles written about this very topic. I just wanted to share my personal experience with using MongoDB in Zend Framework using the Morph library.
The most painful part of programming for me is the CRUD of persistant data storage. I can’t think of anything more tedious than creating complex SQL schemas, modelling my data, creating getters and setters etc. I know what data I need to store, retrieve and manipulate and I just want to dive into the code without having to waste hours setting up the database. But the real pain starts during iterations where I have to update the schema and deploy the changes to all environments – or worse, roll back schema changes.
After discovering Django‘s modelling layer I realised what I was missing out on. It made perfect sense to me. Make your models first and have the SQL managed by the framework automatically. Of course this was still quite annoying because model/schema changes were quite difficult to do but at least there was no need for CRUD because you get it all for free. No more complex join queries!
Once I started using Google App Engine‘s Datastore I was free of SQL’s constraints. I created my data models and if I needed to add an extra field to my model I could do so without having to worry about running ALTER TABLE commands.
I’ve spent the last 2 years working with another noSQL system, CouchDB and again, I’m not here to compare the two but MongoDB just seems more suitable for my needs. I have now integrated MongoDB into my Zend Framework projects and it’s hard to imagine how I ever lived without it. It’s ridiculously easy to set up, no CRUD and it’s incredibly intuitive. Here’s how:
1. Download and run MongoDB service
Once you have downloaded the binaries, create a directory where you want your data stored. Remember MongoDB creates individual BSON (binary JSON) documents for each record. You can start the service by running:
shell> mongod --dbpath=/path/to/my/mongodata
2. Install the MongoDB PHP extension
After downloading the extension install and add to your php.ini
3. Download the Morph library
Add the Morph lib to your PHP include path (or autoload in ZF)
You are ready to go. Let’s try and use it in a simple ZF project. Let’s say we want to create a shopping cart that holds a product name, quantity, price and colour.
Create a “Cart.php” class in /models directory and extend Morph_Object.
Believe it or not, you’ve just finished modelling your data and CRUD and you’re ready to store and retrieve data immediately without having to run any SQL generation scripts. Brilliant.
To store data in say a controller, you would do something like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// instantiate mongo and select your db$mongo=new Mongo();
Morph_Storage::init($mongo->selectDB('shoppingDB'));// instantiate your cart $cart=new Application_Model_Cart();// add data$car->title='Something cool';$cart->quantity=12;$cart->price=15.50;$cart->color="Red";// save the data$cart->save();
You’ve just saved your first record and didn’t have to create individual getters and setters for your fields.
Getting the data out is just as simple:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// instantiate mongo and select your db$mongo=new Mongo();
Morph_Storage::init($mongo->selectDB('shoppingDB'));// instantiate your cart $cart=new Application_Model_Cart();// create a query$query=new Morph_Query();$query->property('price')->greaterThan(1.0);// find records matching query$result=$cart->findByQuery($query);// send results down to view layer $this->view->result=$result;
Check the documentation for the full list of query options. Now loop through the results in your view layer and output the data:
The best part is that since our data store is schema-less, you are free to manipulate your data structure like adding a new field. You do this by simply adding a new property to the Cart class. That’s it.
I hope this post demonstrates the benefits of using noSQL in your PHP projects. Naturally you can use the above in any PHP framework. I just used ZF as an example.
I must have spent over a week coming up with a good solution for this. The effect I was after was a scrollable menu for my game, similar to Angry Birds where the user can flip between levels but also see a preview of the previous/next page on either end of the screen. I came across Nate Murray’s excellent UIScrollView solution but it wasn’t quite what I was looking for and using UIScrollView seemed like too much of an overhead. I needed something simple, preferably wrapped up in a single class.
After searching various community forums I finally came across DK101′s solution which was exactly what I was looking for. A simple CCLayer subclass that would allow me to create an array of CCLayers then add whatever I wanted to it then add the whole lot to my scene. Simple. You are able to add anything you can add to a CCLayer – so a label, menu, image etc. It will all work nicely.
I ended up making a few modifications to it like updating it to work with Cocos2d 0.99.5 and adding the option to set the width of the screens for the Angry Birds style preview effect. There was also an issue with the touch delegate on subsequent scenes. I have put my changes up on Github. Here’s a sample:
To use it in your project:
1. add both files to your project
2. in your scene import CCScrollLayer.h
3. in your scene’s init method construct each layer and pass it to the CCScrollLayer class:
I struggled with this as a n00b in Cocos2d and I finally figured it out so I wanted to share it here in case someone else runs into the same challenge. You may or may not be aware of sprite sheets in traditional 2D games and in CSS where you combine a series of images into a single image, then use x/y offsets to expose the bits of the image you want. Cocos2d makes this process very easy but I wish it was documented better. I’m new to this so please let me know if you think there’s a better way to do it.
This example assumes you know your way around an XCode project and know the basics of Objective-C and Cocos2d. If not, there are plenty of good tutorials around. See my previous post for some useful links. Also, I’m using Cocos2d 0.99.5 so some of this syntax may not work on older versions.
Now, onto the tutorial:
The first thing you want to do is get yourself a sprite sheet and associated plist file. Cocos2d has a bunch of these in its examples folder but you can also make your own using the excellent Zwoptex app. The app will let you add a bunch of images and it will then re-arrange them, export a single image and also generate a plist file containing all the coordinates. This means you won’t have to mess around with x/y pixels to get everything lined up.
Important: make sure in the Zwoptex publish settings you select Cocos2d for the coordinates format.
Once you have added the two files to your project, you can load all the individual frames in the cache and use them however you like.
1
2
// load the plist file into the frame cache[[CCSpriteFrameCache sharedSpriteFrameCache] addSpriteFramesWithFile:@"mysprite.plist"];
That’s it. Zwoptex will give each frame its own name. Look inside the plist file to see what they are called. These will most likely be the original file names of the images before you combined them into one.
Now that the frames are in the cache, we can create and display them as individual sprites like this
You can add any of the other frames to the scene the same way, just make sure you use the correct frame name like dude2.png, dude3.png etc. Again, if you’re not sure what the names are, just open the plist and look at the frames node.
Now that we have all our frames ready to use, we can do some cool things with it like animate them or use them as rollover effects for menu buttons. In the game I’m currently developing I wanted the character to blink from time to time. So I used two sprite images, one with the character’s eyes open and one with its eyes shut. You can then create a sequence and animate them like this:
// Load sprite plist file[[CCSpriteFrameCache sharedSpriteFrameCache] addSpriteFramesWithFile:@"dude_blinking.plist"];
// Create a blank array to store all the frames NSMutableArray*blinkAnimFrames =[NSMutableArray array];
// Add the frames 3 times. One with eyes open, one shut and one open again [blinkAnimFrames addObject:[[CCSpriteFrameCache sharedSpriteFrameCache] spriteFrameByName:[NSString stringWithFormat:@"dude_open.png"]]];
[blinkAnimFrames addObject:[[CCSpriteFrameCache sharedSpriteFrameCache] spriteFrameByName:[NSString stringWithFormat:@"dude_shut.png"]]];
[blinkAnimFrames addObject:[[CCSpriteFrameCache sharedSpriteFrameCache] spriteFrameByName:[NSString stringWithFormat:@"dude_open.png"]]];
// Create a 3-frame animation using the array and use 0.2 seconds delay between each frame
CCAnimation *blinkanimation =[CCAnimation animationWithName:@"blinker" frames:blinkAnimFrames];
CCAnimate *blink =[CCAnimate actionWithDuration:0.2f animation:animation restoreOriginalFrame:NO];
// Now create the blinking action sequence
CCAction *blinkAction =[CCRepeatForever actionWithAction:[CCSequence actions:[CCDelayTime actionWithDuration:2.0f],
blink,
[CCDelayTime actionWithDuration:3.0f],
blink,
[CCDelayTime actionWithDuration:0.2f],
blink,
[CCDelayTime actionWithDuration:2.0f],
nil]];
// Add the actual sprite (first frame from cache) to the scene
CCSprite *dude =[CCSprite spriteWithSpriteFrameName:@"dude_open.png"];
[self addChild: dude];
// Finally, run the blinking animation[dude runAction: blinkAction];
If you study the blinkAction carefully, you’ll see we have just created an endless loop that runs the 3-frame blink action with some pauses in the middle. The sequence is:
(start loop)
1 – Pause for 2 seconds
2 – Blink
3 – Pause for 3 seconds
4 – Blink
5 – Pause for 0.2 seconds
6 – Blink
7 – Pause for 2 seconds
(end loop)
This gives us a semi-realistic blinking animation like this:
There seem to be a lot of developers who like the idea of Test-Driven Development (TDD) and can clearly see the benefit of having tests written for their code but can’t seem to get their head around the process. How do you start writing unit tests before writing the actual code?
Let’s start with an example. You want to write a method that takes a URL as an argument and have it tell you through a boolean return if it’s the correct domain or not. It seems simple enough. Just write your method, pass the URL through some regular expression and you’re done.
But you yourself already know which domains are allowed and which are not so before writing the actual code you can run some tests in your head. E.g. I only want www.bbc.co.uk and its sub-domains on http and https. Nothing else should be allowed. So https://beta.bbc.co.uk/iplayer should return TRUE and http://www.bbcbb.com should return FALSE etc.
The process behind TDD is that you first write a failing test. Then you write the actual code and adjust until the test passes.
So let’s write some tests for our domain checker. I’m using Python and Unittest here:
importunittest# this is what we're going to be testingclass Utils():
def is_bbc(self, val):
pass#placeholder# this is the actual testclass TestUtils(unittest.TestCase):
def setUp(self):
self.u = Utils()def test_is_bbc(self):
self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))self.assertFalse(self.u.is_bbc('http://www.bbc.com'))self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)unittest.TextTestRunner(verbosity=2).run(suite)
I’ve created a an empty method where our domain checker is going to live but as you can see it doesn’t do anything. The tests should immediately make sense. We pass a bunch of domain variations and we know which ones should pass or fail. Naturally, running the test right now will fail:
$ python sample.py
test_is_bbc (__main__.TestUtils) ... FAIL
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)----------------------------------------------------------------------
Traceback (most recent call last):
File "sample.py", line 14, in test_is_bbc
self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))
AssertionError
----------------------------------------------------------------------
Ran 1testin 0.000s
FAILED (failures=1)
Now you can start writing the actual code and keep running the same tests until it passes:
importunittestimportre# this is what we're going to be testingclass Utils():
def is_bbc(self, val):
returnre.match('^https?://([^/]+)?\.bbc\.co\.uk', val)# this is the actual testclass TestUtils(unittest.TestCase):
def setUp(self):
self.u = Utils()def test_is_bbc(self):
self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/iplayer'))self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk/food'))self.assertTrue(self.u.is_bbc('http://www.bbc.co.uk'))self.assertTrue(self.u.is_bbc('https://www.bbc.co.uk'))self.assertTrue(self.u.is_bbc('http://beta.bbc.co.uk'))self.assertFalse(self.u.is_bbc('http://www.bbc.com'))self.assertFalse(self.u.is_bbc('http://www.bbbc.co.uk'))self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))#this should fail
suite = unittest.TestLoader().loadTestsFromTestCase(TestUtils)unittest.TextTestRunner(verbosity=2).run(suite)
The regex may look like it’s correct but running the test will fail again:
$ python sample.py
test_is_bbc (__main__.TestUtils) ... FAIL
======================================================================
FAIL: test_is_bbc (__main__.TestUtils)----------------------------------------------------------------------
Traceback (most recent call last):
File "sample.py", line 22, in test_is_bbc
self.assertFalse(self.u.is_bbc('http://.bbc.co.uk'))
AssertionError
----------------------------------------------------------------------
Ran 1testin 0.001s
FAILED (failures=1)
It has failed on the final assert because our code will also allow http://.bbc.co.uk and we obviously don’t want that. But as you can see we’ve caught this edge case before deploying our app so we can promptly fix our code.
Hopefully this example demonstrates why it’s a good idea to start with tests. This is obviously a simple example but on bigger projects predicting the outcome of your system can save you a lot of debugging time in the future.
