MongoDB in Django

Traditional relational databases (mySQL PostgreSQL etc) and noSQL systems are not mutually exclusive. I have several Django applications that are happily using mySQL. If your site is not scaling due to your database, you are doing it wrong! noSQL will not help you until you start caching some of those expensive queries using something like Memcached.

I use MongoDB alongside mySQL for all the dirty work like storing stats for later processing. There’s no point in polluting mySQL with this sort of data, especially when you’re dealing with millions of entries.

This post is intended for absolute beginners who use Django tranditionally and are curious about how they can integrate a secondary storage service into their apps. I’m assuming you have already installed MongoDB on your dev environment. You will also need to install the MongoEngine library for Python.

Let’s start.

You already know how to create data models in Django, but let’s say we want to store an activity feed for your users everytime they do something on your site. We begin by creating a data model similar to Django’s ORM using MongoEngine but the difference here is that you don’t need to run “syncdb” to create your tables. Mongo’s collections (similar to SQL tables) are schemaless so these models can be manipulated and you won’t need to worry about running migration scripts.

Let’s create a simple collection for storing user activities. Create a file where you normally keep your Django models and call it mongomodel.py

1
2
3
4
5
6
7
8
9
10
11
12

from mongoengine import *
 
# connect to a db (no need to create this - it will be created automagically)
connect('useractivity')
 
class Author(Document):
    pk = IntField()
    name = StringField(max_length=200, required=True)
 
class Activity(Document):
    message = StringField(max_length=200)
    author = ReferenceField(Author, reverse_delete_rule=CASCADE)

“What’s this??!! Django already has a User model, why do I need another in Mongo?” Well, you don’t, but say you want your activity to say something like: “Joe uploaded a photo” and you want Joe’s name to be linked to his profile page. We keep a reference to his mySQL id in case we need to look up other info or construct a URL.

You’ll also notice in the Activity model we are referencing the Author model. This is like a foreign key that will allow us to create relationships, similar to SQL. The CASCADE option will make sure if the user is deleted, all activities are also cleared out.

Ok, let’s start using this puppy! Using the example above we want to create an activity for Joe next time he uploads a photo. First, import mongomodel.py whenever you’re planning to interact with Mongo. In my photo upload view function I will create an activity like so:

1
2
3
4
5
6
7
8
9
10
11

# After photo upload is complete
 
from main.mongomodel import *
 
# first create a user object - you can grab data from request object
the_author = Author(pk=request.user.id, name=request.user.first_name)
the_author.save()
 
# now create the activity
activity = Activity(message='uploaded a new photo', author=the_author)
activity.save()

That’s it. If you decide later you also want to add the name of the file uploaded you can simply add a new field to your Activity model and it will just work, plus it will be backwards compatible, i.e. older records without this field will not complain. Lovely.

Displaying the activity is just as simple. In your view function pull out the record and push down to your template:

1
2
3
4
5
6
7

from main.mongomodel import *
 
# get all activities
activities = Activity.objects
 
# push down to template
return render_to_response('activities.html', {'activities':activites})

Now in your template loop and output like any other model:

1
2
3
4
5

<ul>
{% for a in activites %}
    <li><a href="{% url main.views.profile a.author.pk %}">{{ a.author.name }}</a> {{ a.message }}</li>
{% endfor %}
</ul>

I’ve used the user’s mySQL primary key to construct his profile URL.

This is a very basic example but hopefully you can see the advantage of offloading some of the data storage to Mongo. You may ask “but what if the user changes his name? won’t the data in the activity remain out of sync?”. Yes, it will, but you can very easily add a simple method in your Django user model to update Mongo records whenever the user’s details are updated.

Good luck.

Zend Framework + MongoDB + Morph

This is not another SQL vs noSQL rant. I’m not here to defend one or the other and there are plenty of articles written about this very topic. I just wanted to share my personal experience with using MongoDB in Zend Framework using the Morph library.

The most painful part of programming for me is the CRUD of persistant data storage. I can’t think of anything more tedious than creating complex SQL schemas, modelling my data, creating getters and setters etc. I know what data I need to store, retrieve and manipulate and I just want to dive into the code without having to waste hours setting up the database. But the real pain starts during iterations where I have to update the schema and deploy the changes to all environments – or worse, roll back schema changes.

After discovering Django‘s modelling layer I realised what I was missing out on. It made perfect sense to me. Make your models first and have the SQL managed by the framework automatically. Of course this was still quite annoying because model/schema changes were quite difficult to do but at least there was no need for CRUD because you get it all for free. No more complex join queries!

Once I started using Google App Engine‘s Datastore I was free of SQL’s constraints. I created my data models and if I needed to add an extra field to my model I could do so without having to worry about running ALTER TABLE commands.

I’ve spent the last 2 years working with another noSQL system, CouchDB and again, I’m not here to compare the two but MongoDB just seems more suitable for my needs. I have now integrated MongoDB into my Zend Framework projects and it’s hard to imagine how I ever lived without it. It’s ridiculously easy to set up, no CRUD and it’s incredibly intuitive. Here’s how:

1. Download and run MongoDB service
Once you have downloaded the binaries, create a directory where you want your data stored. Remember MongoDB creates individual BSON (binary JSON) documents for each record. You can start the service by running:

shell> mongod --dbpath=/path/to/my/mongodata

2. Install the MongoDB PHP extension
After downloading the extension install and add to your php.ini

3. Download the Morph library
Add the Morph lib to your PHP include path (or autoload in ZF)

You are ready to go. Let’s try and use it in a simple ZF project. Let’s say we want to create a shopping cart that holds a product name, quantity, price and colour.

Create a “Cart.php” class in /models directory and extend Morph_Object.

1
2
3
4
5
6
7
8
9
10
11
12
13

class Application_Model_Cart extends Morph_Object
{
 
    public function __construct($id = null)
    {      
        parent::__construct($id);
        $this->addProperty(new Morph_Property_String('title'))
             ->addProperty(new Morph_Property_Integer('quantity'))
             ->addProperty(new Morph_Property_Float('price'))
             ->addProperty(new Morph_Property_String('colour'));     
    }
 
}

Believe it or not, you’ve just finished modelling your data and CRUD and you’re ready to store and retrieve data immediately without having to run any SQL generation scripts. Brilliant.

To store data in say a controller, you would do something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

// instantiate mongo and select your db
$mongo = new Mongo();
Morph_Storage::init($mongo->selectDB('shoppingDB'));
 
// instantiate your cart        
$cart = new Application_Model_Cart();
 
// add data
$car->title = 'Something cool';
$cart->quantity = 12;
$cart->price = 15.50;
$cart->color = "Red";
 
// save the data
$cart->save();

You’ve just saved your first record and didn’t have to create individual getters and setters for your fields.

Getting the data out is just as simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// instantiate mongo and select your db
$mongo = new Mongo();
Morph_Storage::init($mongo->selectDB('shoppingDB'));
 
// instantiate your cart        
$cart = new Application_Model_Cart();
 
// create a query
$query = new Morph_Query();
$query->property('price')->greaterThan(1.0);
 
// find records matching query
$result = $cart->findByQuery($query);
 
// send results down to view layer        
$this->view->result = $result;

Check the documentation for the full list of query options. Now loop through the results in your view layer and output the data:

1
2
3

<? foreach($this->result as $item): ?>
    <li><?=$item->title?></li>
<? endforeach ?>

The best part is that since our data store is schema-less, you are free to manipulate your data structure like adding a new field. You do this by simply adding a new property to the Cart class. That’s it.

I hope this post demonstrates the benefits of using noSQL in your PHP projects. Naturally you can use the above in any PHP framework. I just used ZF as an example.

Happy noSQL coding!