This document describes Celery 2.4. For development docs, go here.
Configuration and defaults¶
This document describes the configuration options available.
If you’re using the default loader, you must create the celeryconfig.py
module and make sure it is available on the Python path.
- Example configuration file
- Configuration Directives
- Concurrency settings
- Task result backend settings
- Database backend settings
- AMQP backend settings
- Cache backend settings
- Tokyo Tyrant backend settings
- Redis backend settings
- MongoDB backend settings
- Message Routing
- Broker Settings
- Task execution settings
- Worker: celeryd
- Error E-Mails
- Events
- Broadcast Commands
- Logging
- Custom Component Classes (advanced)
- Periodic Task Server: celerybeat
- Monitor Server: celerymon
- Deprecated Settings
Example configuration file¶
This is an example configuration file to get you started. It should contain all you need to run a basic Celery set-up.
# List of modules to import when celery starts.
CELERY_IMPORTS = ("myapp.tasks", )
## Result store settings.
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "sqlite:///mydatabase.db"
## Broker settings.
BROKER_URL = "amqp://guest:guest@localhost:5672//"
## Worker settings
## If you're doing mostly I/O you can have more processes,
## but if mostly spending CPU, try to keep it close to the
## number of CPUs on your machine. If not set, the number of CPUs/cores
## available will be used.
CELERYD_CONCURRENCY = 10
Configuration Directives¶
Concurrency settings¶
CELERYD_CONCURRENCY¶
The number of concurrent worker processes/threads/green threads, executing tasks.
Defaults to the number of available CPUs.
CELERYD_PREFETCH_MULTIPLIER¶
How many messages to prefetch at a time multiplied by the number of concurrent processes. The default is 4 (four messages for each process). The default setting is usually a good choice, however – if you have very long running tasks waiting in the queue and you have to start the workers, note that the first worker to start will receive four times the number of messages initially. Thus the tasks may not be fairly distributed to the workers.
Task result backend settings¶
CELERY_RESULT_BACKEND¶
Deprecated aliases: | |
---|---|
CELERY_BACKEND |
The backend used to store task results (tombstones). Disabled by default. Can be one of the following:
- database
Use a relational database supported by SQLAlchemy. See Database backend settings.
- cache
Use memcached to store the results. See Cache backend settings.
- mongodb
Use MongoDB to store the results. See MongoDB backend settings.
- redis
Use Redis to store the results. See Redis backend settings.
- tyrant
Use Tokyo Tyrant to store the results. See Tokyo Tyrant backend settings.
- amqp
Send results back as AMQP messages See AMQP backend settings.
CELERY_RESULT_SERIALIZER¶
Result serialization format. Default is “pickle”. See Serializers for information about supported serialization formats.
Database backend settings¶
CELERY_RESULT_DBURI¶
Please see Supported Databases for a table of supported databases. To use this backend you need to configure it with an Connection String, some examples include:
# sqlite (filename)
CELERY_RESULT_DBURI = "sqlite:///celerydb.sqlite"
# mysql
CELERY_RESULT_DBURI = "mysql://scott:tiger@localhost/foo"
# postgresql
CELERY_RESULT_DBURI = "postgresql://scott:tiger@localhost/mydatabase"
# oracle
CELERY_RESULT_DBURI = "oracle://scott:tiger@127.0.0.1:1521/sidname"
See Connection String for more information about connection strings.
CELERY_RESULT_ENGINE_OPTIONS¶
To specify additional SQLAlchemy database engine options you can use
the CELERY_RESULT_ENGINE_OPTIONS
setting:
# echo enables verbose logging from SQLAlchemy.
CELERY_RESULT_ENGINE_OPTIONS = {"echo": True}
Short lived sessions are disabled by default. If enabled they can drastically reduce performance, especially on systems processing lots of tasks. This option is useful on low-traffic workers that experience errors as a result of cached database connections going stale through inactivity. For example, intermittent errors like (OperationalError) (2006, ‘MySQL server has gone away’) can be fixed by enabling short lived sessions. This option only affects the database backend.
Example configuration¶
CELERY_RESULT_BACKEND = "database"
CELERY_RESULT_DBURI = "mysql://user:password@host/dbname"
AMQP backend settings¶
Note
The AMQP backend requires RabbitMQ 1.1.0 or higher to automatically expire results. If you are running an older version of RabbitmQ you should disable result expiration like this:
CELERY_TASK_RESULT_EXPIRES = None
CELERY_RESULT_EXCHANGE¶
Name of the exchange to publish results in. Default is “celeryresults”.
CELERY_RESULT_EXCHANGE_TYPE¶
The exchange type of the result exchange. Default is to use a direct exchange.
CELERY_RESULT_PERSISTENT¶
If set to True
, result messages will be persistent. This means the
messages will not be lost after a broker restart. The default is for the
results to be transient.
Example configuration¶
CELERY_RESULT_BACKEND = "amqp"
CELERY_TASK_RESULT_EXPIRES = 18000 # 5 hours.
Cache backend settings¶
Note
The cache backend supports the pylibmc and python-memcached libraries. The latter is used only if pylibmc is not installed.
CELERY_CACHE_BACKEND¶
Using a single memcached server:
CELERY_CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
Using multiple memcached servers:
CELERY_RESULT_BACKEND = "cache"
CELERY_CACHE_BACKEND = 'memcached://172.19.26.240:11211;172.19.26.242:11211/'
The “dummy” backend stores the cache in memory only:
CELERY_CACHE_BACKEND = “dummy”
CELERY_CACHE_BACKEND_OPTIONS¶
You can set pylibmc options using the CELERY_CACHE_BACKEND_OPTIONS
setting:
CELERY_CACHE_BACKEND_OPTIONS = {"binary": True,
"behaviors": {"tcp_nodelay": True}}
Tokyo Tyrant backend settings¶
Note
The Tokyo Tyrant backend requires the pytyrant
library:
http://pypi.python.org/pypi/pytyrant/
This backend requires the following configuration directives to be set:
TT_HOST¶
Host name of the Tokyo Tyrant server.
TT_PORT¶
The port the Tokyo Tyrant server is listening to.
Example configuration¶
CELERY_RESULT_BACKEND = "tyrant"
TT_HOST = "localhost"
TT_PORT = 1978
Redis backend settings¶
Note
The Redis backend requires the redis
library:
http://pypi.python.org/pypi/redis/
To install the redis package use pip or easy_install:
$ pip install redis
This backend requires the following configuration directives to be set.
CELERY_REDIS_HOST¶
Host name of the Redis database server. e.g. “localhost”.
CELERY_REDIS_PORT¶
Port to the Redis database server. e.g. 6379.
CELERY_REDIS_DB¶
Database number to use. Default is 0
CELERY_REDIS_PASSWORD¶
Password used to connect to the database.
Example configuration¶
CELERY_RESULT_BACKEND = "redis"
CELERY_REDIS_HOST = "localhost"
CELERY_REDIS_PORT = 6379
CELERY_REDIS_DB = 0
MongoDB backend settings¶
Note
The MongoDB backend requires the pymongo
library:
http://github.com/mongodb/mongo-python-driver/tree/master
CELERY_MONGODB_BACKEND_SETTINGS¶
This is a dict supporting the following keys:
- host
Host name of the MongoDB server. Defaults to “localhost”.
- port
The port the MongoDB server is listening to. Defaults to 27017.
- user
User name to authenticate to the MongoDB server as (optional).
- password
Password to authenticate to the MongoDB server (optional).
- database
The database name to connect to. Defaults to “celery”.
- taskmeta_collection
The collection name to store task meta data. Defaults to “celery_taskmeta”.
Example configuration¶
CELERY_RESULT_BACKEND = "mongodb"
CELERY_MONGODB_BACKEND_SETTINGS = {
"host": "192.168.1.100",
"port": 30000,
"database": "mydb",
"taskmeta_collection": "my_taskmeta_collection",
}
Message Routing¶
CELERY_QUEUES¶
The mapping of queues the worker consumes from. This is a dictionary of queue name/options. See Routing Tasks for more information.
The default is a queue/exchange/binding key of “celery”, with exchange type direct.
You don’t have to care about this unless you want custom routing facilities.
CELERY_ROUTES¶
A list of routers, or a single router used to route tasks to queues. When deciding the final destination of a task the routers are consulted in order. See Routers for more information.
CELERY_CREATE_MISSING_QUEUES¶
If enabled (default), any queues specified that is not defined in
CELERY_QUEUES
will be automatically created. See
Automatic routing.
CELERY_DEFAULT_QUEUE¶
The queue used by default, if no custom queue is specified. This queue must
be listed in CELERY_QUEUES
. The default is: celery.
CELERY_DEFAULT_EXCHANGE¶
Name of the default exchange to use when no custom exchange is specified. The default is: celery.
CELERY_DEFAULT_EXCHANGE_TYPE¶
Default exchange type used when no custom exchange is specified. The default is: direct.
CELERY_DEFAULT_ROUTING_KEY¶
The default routing key used when sending tasks. The default is: celery.
CELERY_DEFAULT_DELIVERY_MODE¶
Can be transient or persistent. The default is to send persistent messages.
Broker Settings¶
BROKER_TRANSPORT¶
Aliases: | BROKER_BACKEND |
---|---|
Deprecated aliases: | |
CARROT_BACKEND |
The Kombu transport to use. Default is amqplib
.
You can use a custom transport class name, or select one of the
built-in transports: amqplib
, pika
, redis
, beanstalk
,
sqlalchemy
, django
, mongodb
, couchdb
.
BROKER_URL¶
Default broker URL. This must be an URL in the form of:
transport://userid:password@hostname:port/virtual_host
Only the scheme part (transport://
) is required, the rest
is optional, and defaults to the specific transports default values.
If this setting is defined it will override a subset of the
other BROKER
options. These options are BROKER_HOST
,
BROKER_USER
, BROKER_PASSWORD
, BROKER_PORT
,
and BROKER_VHOST
.
See the Kombu documentation for more information about broker URLs.
BROKER_HOST¶
Hostname of the broker.
BROKER_PORT¶
Custom port of the broker. Default is to use the default port for the selected backend.
BROKER_USER¶
Username to connect as.
BROKER_PASSWORD¶
Password to connect with.
BROKER_VHOST¶
Virtual host. Default is “/”.
BROKER_USE_SSL¶
Use SSL to connect to the broker. Off by default. This may not be supported by all transports.
BROKER_POOL_LIMIT¶
New in version 2.3.
The maximum number of connections that can be open in the connection pool.
A good default value could be 10, or more if you’re using eventlet/gevent or lots of threads.
If set to None
or 0 the connection pool will be disabled and
connections will be established and closed for every use.
Disabled by default.
BROKER_CONNECTION_TIMEOUT¶
The default timeout in seconds before we give up establishing a connection to the AMQP server. Default is 4 seconds.
BROKER_CONNECTION_RETRY¶
Automatically try to re-establish the connection to the AMQP broker if lost.
The time between retries is increased for each retry, and is
not exhausted before BROKER_CONNECTION_MAX_RETRIES
is
exceeded.
This behavior is on by default.
BROKER_CONNECTION_MAX_RETRIES¶
Maximum number of retries before we give up re-establishing a connection to the AMQP broker.
If this is set to 0
or None
, we will retry forever.
Default is 100 retries.
BROKER_TRANSPORT_OPTIONS¶
New in version 2.2.
A dict of additional options passed to the underlying transport.
See your transport user manual for supported options (if any).
Task execution settings¶
CELERY_ALWAYS_EAGER¶
If this is True
, all tasks will be executed locally by blocking until
the task returns. apply_async()
and Task.delay()
will return
an EagerResult
instance, which emulates the API
and behavior of AsyncResult
, except the result
is already evaluated.
That is, tasks will be executed locally instead of being sent to the queue.
CELERY_EAGER_PROPAGATES_EXCEPTIONS¶
If this is True
, eagerly executed tasks (applied by task.apply(),
or when the CELERY_ALWAYS_EAGER
setting is enabled), will
propagate exceptions.
It’s the same as always running apply()
with throw=True
.
CELERY_IGNORE_RESULT¶
Whether to store the task return values or not (tombstones).
If you still want to store errors, just not successful return values,
you can set CELERY_STORE_ERRORS_EVEN_IF_IGNORED
.
CELERY_MESSAGE_COMPRESSION¶
Default compression used for task messages.
Can be "gzip"
, "bzip2"
(if available), or any custom
compression schemes registered in the Kombu compression registry.
The default is to send uncompressed messages.
CELERY_TASK_RESULT_EXPIRES¶
Time (in seconds, or a timedelta
object) for when after
stored task tombstones will be deleted.
A built-in periodic task will delete the results after this time
(celery.task.backend_cleanup
).
Note
For the moment this only works with the amqp, database, cache, redis and MongoDB backends.
When using the database or MongoDB backends, celerybeat must be running for the results to be expired.
CELERY_MAX_CACHED_RESULTS¶
Result backends caches ready results used by the client.
This is the total number of results to cache before older results are evicted. The default is 5000.
CELERY_TRACK_STARTED¶
If True
the task will report its status as “started” when the
task is executed by a worker. The default value is False
as
the normal behaviour is to not report that level of granularity. Tasks
are either pending, finished, or waiting to be retried. Having a “started”
state can be useful for when there are long running tasks and there is a
need to report which task is currently running.
CELERY_TASK_SERIALIZER¶
A string identifying the default serialization method to use. Can be
pickle (default), json, yaml, msgpack or any custom serialization
methods that have been registered with kombu.serialization.registry
.
See also
CELERY_TASK_PUBLISH_RETRY¶
New in version 2.2.
Decides if publishing task messages will be retried in the case
of connection loss or other connection errors.
See also CELERY_TASK_PUBLISH_RETRY_POLICY
.
Disabled by default.
CELERY_TASK_PUBLISH_RETRY_POLICY¶
New in version 2.2.
Defines the default policy when retrying publishing a task message in the case of connection loss or other connection errors.
This is a mapping that must contain the following keys:
max_retries
Maximum number of retries before giving up, in this case the exception that caused the retry to fail will be raised.
A value of 0 or
None
means it will retry forever.The default is to retry 3 times.
interval_start
Defines the number of seconds (float or integer) to wait between retries. Default is 0, which means the first retry will be instantaneous.
interval_step
On each consecutive retry this number will be added to the retry delay (float or integer). Default is 0.2.
interval_max
Maximum number of seconds (float or integer) to wait between retries. Default is 0.2.
With the default policy of:
{"max_retries": 3,
"interval_start": 0,
"interval_step": 0.2,
"interval_max": 0.2}
the maximum time spent retrying will be 0.4 seconds. It is set relatively short by default because a connection failure could lead to a retry pile effect if the broker connection is down: e.g. many web server processes waiting to retry blocking other incoming requests.
CELERY_DEFAULT_RATE_LIMIT¶
The global default rate limit for tasks.
This value is used for tasks that does not have a custom rate limit The default is no rate limit.
CELERY_DISABLE_RATE_LIMITS¶
Disable all rate limits, even if tasks has explicit rate limits set.
CELERY_ACKS_LATE¶
Late ack means the task messages will be acknowledged after the task has been executed, not just before, which is the default behavior.
See also
Worker: celeryd¶
CELERY_IMPORTS¶
A sequence of modules to import when the celery daemon starts.
This is used to specify the task modules to import, but also to import signal handlers and additional remote control commands, etc.
CELERYD_MAX_TASKS_PER_CHILD¶
Maximum number of tasks a pool worker process can execute before it’s replaced with a new one. Default is no limit.
CELERYD_TASK_TIME_LIMIT¶
Task hard time limit in seconds. The worker processing the task will be killed and replaced with a new one when this is exceeded.
CELERYD_TASK_SOFT_TIME_LIMIT¶
Task soft time limit in seconds.
The SoftTimeLimitExceeded
exception will be
raised when this is exceeded. The task can catch this to
e.g. clean up before the hard time limit comes.
Example:
from celery.task import task
from celery.exceptions import SoftTimeLimitExceeded
@task()
def mytask():
try:
return do_work()
except SoftTimeLimitExceeded:
cleanup_in_a_hurry()
CELERY_STORE_ERRORS_EVEN_IF_IGNORED¶
If set, the worker stores all task errors in the result store even if
Task.ignore_result
is on.
CELERYD_STATE_DB¶
Name of the file used to stores persistent worker state (like revoked tasks). Can be a relative or absolute path, but be aware that the suffix .db may be appended to the file name (depending on Python version).
Can also be set via the --statedb
argument to
celeryd
.
Not enabled by default.
CELERYD_ETA_SCHEDULER_PRECISION¶
Set the maximum time in seconds that the ETA scheduler can sleep between rechecking the schedule. Default is 1 second.
Setting this value to 1 second means the schedulers precision will be 1 second. If you need near millisecond precision you can set this to 0.1.
Error E-Mails¶
CELERY_SEND_TASK_ERROR_EMAILS¶
The default value for the Task.send_error_emails attribute, which if
set to True
means errors occurring during task execution will be
sent to ADMINS
by email.
ADMINS¶
List of (name, email_address) tuples for the administrators that should receive error emails.
SERVER_EMAIL¶
The email address this worker sends emails from. Default is celery@localhost.
EMAIL_HOST¶
The mail server to use. Default is “localhost”.
EMAIL_HOST_USER¶
User name (if required) to log on to the mail server with.
EMAIL_HOST_PASSWORD¶
Password (if required) to log on to the mail server with.
EMAIL_PORT¶
The port the mail server is listening on. Default is 25.
EMAIL_USE_SSL¶
Use SSL when connecting to the SMTP server. Disabled by default.
EMAIL_USE_TLS¶
Use TLS when connecting to the SMTP server. Disabled by default.
EMAIL_TIMEOUT¶
Timeout in seconds for when we give up trying to connect to the SMTP server when sending emails.
The default is 2 seconds.
Example E-Mail configuration¶
This configuration enables the sending of error emails to george@vandelay.com and kramer@vandelay.com:
# Enables error emails.
CELERY_SEND_TASK_ERROR_EMAILS = True
# Name and email addresses of recipients
ADMINS = (
("George Costanza", "george@vandelay.com"),
("Cosmo Kramer", "kosmo@vandelay.com"),
)
# Email address used as sender (From field).
SERVER_EMAIL = "no-reply@vandelay.com"
# Mailserver configuration
EMAIL_HOST = "mail.vandelay.com"
EMAIL_PORT = 25
# EMAIL_HOST_USER = "servers"
# EMAIL_HOST_PASSWORD = "s3cr3t"
Events¶
CELERY_SEND_EVENTS¶
Send events so the worker can be monitored by tools like celerymon.
CELERY_SEND_TASK_SENT_EVENT¶
New in version 2.2.
If enabled, a task-sent event will be sent for every task so tasks can be tracked before they are consumed by a worker.
Disabled by default.
CELERY_EVENT_SERIALIZER¶
Message serialization format used when sending event messages. Default is “json”. See Serializers.
Broadcast Commands¶
CELERY_BROADCAST_QUEUE¶
Name prefix for the queue used when listening for broadcast messages. The workers host name will be appended to the prefix to create the final queue name.
Default is “celeryctl”.
CELERY_BROADCAST_EXCHANGE¶
Name of the exchange used for broadcast messages.
Default is “celeryctl”.
CELERY_BROADCAST_EXCHANGE_TYPE¶
Exchange type used for broadcast messages. Default is “fanout”.
Logging¶
CELERYD_HIJACK_ROOT_LOGGER¶
New in version 2.2.
By default any previously configured logging options will be reset, because the Celery programs “hijacks” the root logger.
If you want to customize your own logging then you can disable this behavior.
Note
Logging can also be customized by connecting to the
celery.signals.setup_logging
signal.
CELERYD_LOG_COLOR¶
Enables/disables colors in logging output by the Celery apps.
By default colors are enabled if
- the app is logging to a real terminal, and not a file.
- the app is not running on Windows.
CELERYD_LOG_FORMAT¶
The format to use for log messages.
Default is [%(asctime)s: %(levelname)s/%(processName)s] %(message)s
See the Python logging
module for more information about log
formats.
CELERYD_TASK_LOG_FORMAT¶
The format to use for log messages logged in tasks. Can be overridden using
the --loglevel
option to celeryd
.
Default is:
[%(asctime)s: %(levelname)s/%(processName)s]
[%(task_name)s(%(task_id)s)] %(message)s
See the Python logging
module for more information about log
formats.
CELERY_REDIRECT_STDOUTS¶
If enabled stdout and stderr will be redirected to the current logger.
Enabled by default. Used by celeryd and celerybeat.
CELERY_REDIRECT_STDOUTS_LEVEL¶
The log level output to stdout and stderr is logged as.
Can be one of DEBUG
, INFO
, WARNING
,
ERROR
or CRITICAL
.
Default is WARNING
.
Custom Component Classes (advanced)¶
CELERYD_POOL¶
Name of the pool class used by the worker.
You can use a custom pool class name, or select one of
the built-in aliases: processes
, eventlet
, gevent
.
Default is processes
.
CELERYD_AUTOSCALER¶
New in version 2.2.
Name of the autoscaler class to use.
Default is "celery.worker.autoscale.Autoscaler"
.
CELERYD_CONSUMER¶
Name of the consumer class used by the worker.
Default is celery.worker.consumer.Consumer
CELERYD_MEDIATOR¶
Name of the mediator class used by the worker.
Default is celery.worker.controllers.Mediator
.
CELERYD_ETA_SCHEDULER¶
Name of the ETA scheduler class used by the worker.
Default is celery.utils.timer2.Timer
, or one overrided
by the pool implementation.
Periodic Task Server: celerybeat¶
CELERYBEAT_SCHEDULE¶
The periodic task schedule used by celerybeat
.
See Entries.
CELERYBEAT_SCHEDULER¶
The default scheduler class. Default is “celery.beat.PersistentScheduler”.
Can also be set via the -S
argument to
celerybeat
.
CELERYBEAT_SCHEDULE_FILENAME¶
Name of the file used by PersistentScheduler to store the last run times of periodic tasks. Can be a relative or absolute path, but be aware that the suffix .db may be appended to the file name (depending on Python version).
Can also be set via the --schedule
argument to
celerybeat
.
CELERYBEAT_MAX_LOOP_INTERVAL¶
The maximum number of seconds celerybeat
can sleep
between checking the schedule. Default is 300 seconds (5 minutes).
Deprecated Settings¶
These settings have been deprecated and should no longer used, as they will be removed in future versions.
CELERY_AMQP_TASK_RESULT_EXPIRES¶
Deprecated since version 2.5.
The time in seconds of which the task result queues should expire.
This setting is deprecated, and will be removed in version 3.0.
Please use CELERY_TASK_RESULT_EXPIRES
instead.
Note
AMQP result expiration requires RabbitMQ versions 2.1.0 and higher.
CELERY_TASK_ERROR_WHITELIST¶
Deprecated since version 2.5.
A white list of exceptions to send error emails for.
This option is pending deprecation and is scheduled for removal in version 3.0.
CELERYD_LOG_FILE¶
Deprecated since version 2.4.
This option is deprecated and is scheduled for removal in version 3.0.
Please use the --logfile
argument instead.
The default file name the worker daemon logs messages to. Can be overridden
using the --logfile
option to celeryd
.
The default is None
(stderr)
CELERYD_LOG_LEVEL¶
Deprecated since version 2.4.
This option is deprecated and is scheduled for removal in version 3.0.
Please use the --loglevel
argument instead.
Worker log level, can be one of DEBUG
, INFO
, WARNING
,
ERROR
or CRITICAL
.
Can also be set via the --loglevel
argument to
celeryd
.
See the logging
module for more information.
CELERYBEAT_LOG_FILE¶
Deprecated since version 2.4.
This option is deprecated and is scheduled for removal in version 3.0.
Please use the --logfile
argument instead.
The default file name to log messages to. Can be overridden using
the –logfile option to celerybeat
.
The default is None
(stderr).
CELERYBEAT_LOG_LEVEL¶
Deprecated since version 2.4.
This option is deprecated and is scheduled for removal in version 3.0.
Please use the --loglevel
argument instead.
Logging level. Can be any of DEBUG
, INFO
, WARNING
,
ERROR
, or CRITICAL
.
Can also be set via the --loglevel
argument to
celerybeat
.
See the logging
module for more information.
CELERYMON_LOG_FILE¶
Deprecated since version 2.4.
This option is deprecated and is scheduled for removal in version 3.0.
Please use the --logfile
argument instead.
The default file name to log messages to. Can be overridden using
the --logfile
argument to celerymon.
The default is None
(stderr)