*Note: This is a recap of a lightning talk I gave at [DjangoCon 2009] [djangocon]. Here are [the slides](http://www.slideshare.net/cmheisel/gearman-and-django-working-later)*
Problem: You’ve got resource or time intensive work to be done but you don’t want to do it within the Django request/response cycle. Jobs like fetching remote resources, resizing images, expensive database queries, etc. are good candidates.
At work, whenever we have a problem with scaling, or really any problem at all, [Johnny][johnny] always asks **”What would [Brad Fitzpatrick][brad] do?”**
What he did was build [Gearman][gearman]. It is a work queuing and distribution system. From your client code you send a request off for work to be done, either synchronously or asynchronously. Worker instances pull jobs off the stack and perform them.
The Python bindings are [stupidly simple to use](http://samuel.github.com/python-gearman/docs/).
# client.py from gearman import GearmanClient client = GearmanClient(["127.0.0.1"]) client.dispatch_background_task("echo", "foo") # worker.py worker = GearmanWorker(["127.0.0.1"]) worker.register_function("echo", lambda job:job.arg) worker.work()
It’s conceptually similar to [RabbitMQ][rabbitmq]. Both are message queues but Gearman feels a little more specialized for use as a task queue, while RabbitMQ is more a general messaging framework.
RabbitMQ is nice as well but I’ve got one particular pet peeve with it. When you ask “How do I scale RabbitMQ?” the answer is pretty much [“Use the power of Erlang”][rabbit-scale].
When you ask “How do I scale Gearman?” the answer is “just like memcached, add more servers to the cluster and specify them in your connection.”
from gearman import GearmanClient client = GearmanClient(["workserver.yourdomain.com", "procrastinate.yourdomain.com"])
If you’re looking for a task queue, I highly recommend [Gearman][gearman] — it’s easy to use and easy to scale!