Celery is a distributed queue that is very easy to pick-up. I’ll do two quick examples: one that sends a job and returns and another that sends a job and then retrieves a result. I’m going to use SQLite for this example (which is interfaced via SQLAlchemy). Since Celery seems to have some issues importing SQLite under Python 3, we’ll use Python 2.7 . Make sure that you install the “celery” and “sqlalchemy” Python packages.
Without Results
Define the tasks module and save it as sqlite_queue_without_results.py:
import celery
_BACKEND_URI = 'sqla+sqlite:///tasks.sqlite'
_APP = celery.Celery('sqlite_queue_without_results', broker=_BACKEND_URI)
@_APP.task
def some_task(incoming_message):
return "ECHO: {0}".format(incoming_message)
Start the server:
$ celery -A sqlite_queue_without_results worker --loglevel=info
Execute the following Python to submit a job:
import sqlite_queue_without_results
_ARG1 = "Test message"
sqlite_queue_without_results.some_task.delay(_ARG1)
That’s it. You’ll see something similar to the following in the server window:

With Results
This time, when we define the tasks module, we’ll provide Celery a results backend. Call this module sqlite_queue_with_results.py:
import celery
_RESULT_URI = 'db+sqlite:///results.sqlite'
_BACKEND_URI = 'sqla+sqlite:///tasks.sqlite'
_APP = celery.Celery('sqlite_queue_with_results', broker=_BACKEND_URI, backend=_RESULT_URI)
@_APP.task
def some_task(incoming_message):
return "ECHO: {0}".format(incoming_message)
Start the server:
$ celery -A sqlite_queue_with_results worker --loglevel=info
Execute the following Python to submit a job:
import sqlite_queue_with_results
_ARG1 = "Test message"
r = sqlite_queue_with_results.some_task.delay(_ARG1)
value = r.get(timeout=2)
print("Result: [{0}]".format(value))
Since we’re using a traditional DBMS (albeit a fast, local one) to store our results, we’ll be internally polling for a state change and then fetching the result. Therefore, it is a more costly operation and I’ve used a two-second timeout to accommodate this.
The server output will be similar to the following:

The client output will look like:
Result: [ECHO: Test message]
Celery has many more features not explored by this tutorial, including:
- exception propagation
- custom task states (including providing metadata that can be read by the client)
- task ignore/reject/retry responses
- HTTP-based tasks (for calling your tasks in another place or language)
- task routing
- periodic/scheduled tasks
- workflows
- drawing visible graphs in order to inspect behavior
For more information, see the user guide.
Like this:
Like Loading...
You must be logged in to post a comment.