This document describes an older version of Celery (2.5). For the latest stable version please go here.

celery.concurrency.processes

class celery.concurrency.processes.TaskPool(limit=None, putlocks=True, logger=None, **options)

Multiprocessing Pool implementation.

class Pool(processes=None, initializer=None, initargs=(), maxtasksperchild=None, timeout=None, soft_timeout=None, force_execv=False)

Class which supports an async version of the apply() builtin

class Process(*args, **kwargs)
class TaskPool.Pool.ResultHandler(outqueue, get, cache, poll, join_exited_workers, putlock)
body()
exception TaskPool.Pool.SoftTimeLimitExceeded

The soft time limit has been exceeded. This exception is raised to give the task a chance to clean up.

class TaskPool.Pool.Supervisor(pool)
body()
class TaskPool.Pool.TaskHandler(taskqueue, put, outqueue, pool)
body()
class TaskPool.Pool.TimeoutHandler(processes, cache, t_soft, t_hard)
body()
TaskPool.Pool.apply(func, args=(), kwds={})

Equivalent of apply() builtin

TaskPool.Pool.apply_async(func, args=(), kwds={}, callback=None, accept_callback=None, timeout_callback=None, waitforslot=False, error_callback=None, soft_timeout=None, timeout=None)

Asynchronous equivalent of apply() builtin.

Callback is called when the functions return value is ready. The accept callback is called when the job is accepted to be executed.

Simplified the flow is like this:

>>> if accept_callback:
...     accept_callback()
>>> retval = func(*args, **kwds)
>>> if callback:
...     callback(retval)
TaskPool.Pool.close()
TaskPool.Pool.grow(n=1)
TaskPool.Pool.imap(func, iterable, chunksize=1, lost_worker_timeout=10.0)

Equivalent of itertools.imap() – can be MUCH slower than Pool.map()

TaskPool.Pool.imap_unordered(func, iterable, chunksize=1, lost_worker_timeout=10.0)

Like imap() method but ordering of results is arbitrary

TaskPool.Pool.join()
TaskPool.Pool.map(func, iterable, chunksize=None)

Equivalent of map() builtin

TaskPool.Pool.map_async(func, iterable, chunksize=None, callback=None)

Asynchronous equivalent of map() builtin

TaskPool.Pool.restart()
TaskPool.Pool.shrink(n=1)
TaskPool.Pool.terminate()
TaskPool.grow(n=1)
TaskPool.num_processes
TaskPool.on_start()

Run the task pool.

Will pre-fork all workers so they’re ready to accept tasks.

TaskPool.on_stop()

Gracefully stop the pool.

TaskPool.on_terminate()

Force terminate the pool.

TaskPool.requires_mediator = True
TaskPool.restart()
TaskPool.shrink(n=1)
TaskPool.terminate_job(pid, signal=None)
celery.concurrency.processes.WORKER_SIGIGNORE = frozenset(['SIGINT'])

List of signals to ignore when a child process starts.

celery.concurrency.processes.WORKER_SIGRESET = frozenset(['SIGHUP', 'SIGTERM', 'SIGTTOU', 'SIGTTIN', 'SIGUSR1'])

List of signals to reset when a child process starts.

celery.concurrency.processes.process_initializer(app, hostname)

Initializes the process so it can be used to process tasks.

Previous topic

celery.concurrency.solo

Next topic

celery.concurrency.processes.pool

This Page