Optimization and Performance

Table of Contents:



By far the best way to deploy Thorn for optimal web request performance is to use the Celery eventlet/gevent pools. Which one you choose does not matter much, but some will prefer one over the other.

To start a Celery worker with the eventlet/gevent pool set the -P option:

$ celery -A proj worker -l info -P eventlet -c 1000

The -c 1000 tells the worker to use up to one thousand green-threads for task execution.

Note that this will only start one OS process, so to take advantage of multiple CPUs or CPU-cores you need to start multiple processes.

This can be achived by using the CELERYD_NODES option to the Celery generic-init.d script, or by manually starting celery multi, for example if you have four CPU-cores you may want to start four worker instances, with a thousand green-threads each:

$ celery multi start 4 -A proj -P eventlet -c 1000
$ celery multi restart 4 -A proj -P eventlet -c 1000
$ celery multi stop 4 -A proj -P eventlet -c 1000

Eventlet: Asynchronous DNS lookups

To use the Celery eventlet pool you should make sure to install the dnspython library, to enable asynchronous DNS lookups:

$ pip install dnspython

Task retry settings

Prefetch multiplier