Multiprocessing Worker - celery.worker

celery.utils.chunks(it, n)

Split an iterator into chunks with n elements each.

Examples

# n == 2 >>> x = chunks(iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), 2) >>> list(x) [[0, 1], [2, 3], [4, 5], [6, 7], [8, 9], [10]]

# n == 3 >>> x = chunks(iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), 3) >>> list(x) [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 10]]

celery.utils.first(predicate, iterable)

Returns the first element in iterable that predicate returns a True value for.

celery.utils.fun_takes_kwargs(fun, kwlist=[])

With a function, and a list of keyword arguments, returns arguments in the list which the function takes.

If the object has an argspec attribute that is used instead of using the inspect.getargspec`() introspection.

Parameters:

• fun – The function to inspect arguments of.
• kwlist – The list of keyword arguments.

Examples

>>> def foo(self, x, y, logfile=None, loglevel=None):
...     return x * y
>>> fun_takes_kwargs(foo, ["logfile", "loglevel", "task_id"])
["logfile", "loglevel"]

>>> def foo(self, x, y, **kwargs):
>>> fun_takes_kwargs(foo, ["logfile", "loglevel", "task_id"])
["logfile", "loglevel", "task_id"]

celery.utils.gen_unique_id()

Generate a unique id, having - hopefully - a very small chance of collission.

For now this is provided by uuid.uuid4().

celery.utils.get_cls_by_name(name, aliases={})

Get class by name.

The name should be the full dot-separated path to the class:

modulename.ClassName

Example:

celery.worker.pool.TaskPool
                   ^- class name

If aliases is provided, a dict containing short name/long name mappings, the name is looked up in the aliases first.

Examples:

>>> get_cls_by_name("celery.worker.pool.TaskPool")
<class 'celery.worker.pool.TaskPool'>

>>> get_cls_by_name("default", {
...     "default": "celery.worker.pool.TaskPool"})
<class 'celery.worker.pool.TaskPool'>

# Does not try to look up non-string names. >>> from celery.worker.pool import TaskPool >>> get_cls_by_name(TaskPool) is TaskPool True

celery.utils.get_full_cls_name(cls)

With a class, get its full module and class name.

celery.utils.instantiate(name, *args, **kwargs)

Instantiate class by name.

See get_cls_by_name().

celery.utils.mattrgetter(*attrs)

Like operator.itemgetter() but returns None on missing attributes instead of raising AttributeError.

celery.utils.mitemgetter(*items)

Like operator.itemgetter() but returns None on missing items instead of raising KeyError.

celery.utils.noop(*args, **kwargs)

No operation.

Takes any arguments/keyword arguments and does nothing.

celery.utils.padlist(container, size, default=None)

Pad list with default elements.

Examples:

>>> first, last, city = padlist(["George", "Constanza", "NYC"], 3)
("George", "Constanza", "NYC")
>>> first, last, city = padlist(["George", "Constanza"], 3)
("George", "Constanza", None)
>>> first, last, city, planet = padlist(["George", "Constanza",
                                         "NYC"], 4, default="Earth")
("George", "Constanza", "NYC", "Earth")

celery.utils.repeatlast(it)

Iterate over all elements in the iterator, and when its exhausted yield the last value infinitely.

celery.utils.retry_over_time(fun, catch, args=[], kwargs={}, errback=<function noop at 0x47b7d70>, max_retries=None, interval_start=2, interval_step=2, interval_max=30)

Retry the function over and over until max retries is exceeded.

For each retry we sleep a for a while before we try again, this interval is increased for every retry until the max seconds is reached.

Parameters:

• fun – The function to try
• catch – Exceptions to catch, can be either tuple or a single exception class.
• args – Positional arguments passed on to the function.
• kwargs – Keyword arguments passed on to the function.
• errback – Callback for when an exception in catch is raised. The callback must take two arguments: exc and interval, where exc is the exception instance, and interval is the time in seconds to sleep next..
• max_retries – Maximum number of retries before we give up. If this is not set, we will retry forever.
• interval_start – How long (in seconds) we start sleeping between retries.
• interval_step – By how much the interval is increased for each retry.
• interval_max – Maximum number of seconds to sleep between retries.