A short introduction to Django and Gearman

*Note: This is a recap of a lightning talk I gave at [DjangoCon 2009] [djangocon]. Here are [the slides](http://www.slideshare.net/cmheisel/gearman-and-django-working-later)*

Problem: You’ve got resource or time intensive work to be done but you don’t want to do it within the Django request/response cycle. Jobs like fetching remote resources, resizing images, expensive database queries, etc. are good candidates.

At work, whenever we have a problem with scaling, or really any problem at all, [Johnny][johnny] always asks **”What would [Brad Fitzpatrick][brad] do?”**

bradfitz.png

What he did was build [Gearman][gearman]. It is a work queuing and distribution system. From your client code you send a request off for work to be done, either synchronously or asynchronously. Worker instances pull jobs off the stack and perform them.

The Python bindings are [stupidly simple to use](http://samuel.github.com/python-gearman/docs/).

# client.py
from gearman import GearmanClient
client = GearmanClient(["127.0.0.1"])
client.dispatch_background_task("echo", "foo")

# worker.py
worker = GearmanWorker(["127.0.0.1"])
worker.register_function("echo", lambda job:job.arg)
worker.work()

It’s conceptually similar to [RabbitMQ][rabbitmq]. Both are message queues but Gearman feels a little more specialized for use as a task queue, while RabbitMQ is more a general messaging framework.

RabbitMQ is nice as well but I’ve got one particular pet peeve with it. When you ask “How do I scale RabbitMQ?” the answer is pretty much [“Use the power of Erlang”][rabbit-scale].

When you ask “How do I scale Gearman?” the answer is “just like memcached, add more servers to the cluster and specify them in your connection.”

from gearman import GearmanClient

client = GearmanClient(["workserver.yourdomain.com",
"procrastinate.yourdomain.com"])

If you’re looking for a task queue, I highly recommend [Gearman][gearman] — it’s easy to use and easy to scale!

[djangocon]: http://www.djangocon.org/
[brad]: http://bradfitz.com/
[johnny]: http://revjohnnyhealey.wordpress.com/
[gearman]: http://gearman.org/
[rabbitmq]: http://www.rabbitmq.com/
[rabbit-scale]: http://browsertoolkit.com/fault-tolerance.png

About Chris

Python developer, Agile practitioner trying desperately not to be a pointy haired boss.
This entry was posted in Django, Programming, Python. Bookmark the permalink.

2 Responses to A short introduction to Django and Gearman

  1. Zellyn says:

    I didn’t get a chance to say so, but I thought the talk went really well!

  2. Chris says:

    Thanks, I hope it didn’t come off as anti-RabbitMQ/Erlang.

    Their both really nice, but Gearman definitely has they “it just works” factor — if you’re looking specifically for a work queue.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s