This document describes Celery 2.4. For development docs, go here.
Internals: The worker¶
Introduction¶
The worker consists of 4 main components: the consumer, the scheduler, the mediator and the task pool. All these components runs in parallel working with two data structures: the ready queue and the ETA schedule.
Data structures¶
ready_queue¶
The ready queue is either an instance of Queue.Queue
, or
celery.buckets.TaskBucket
. The latter if rate limiting is enabled.
eta_schedule¶
The ETA schedule is a heap queue sorted by time.
Components¶
Consumer¶
Receives messages from the broker using Kombu.
When a message is received it’s converted into a
celery.worker.job.TaskRequest
object.
Tasks with an ETA are entered into the eta_schedule, messages that can be immediately processed are moved directly to the ready_queue.
ScheduleController¶
The schedule controller is running the eta_schedule. If the scheduled tasks eta has passed it is moved to the ready_queue, otherwise the thread sleeps until the eta is met (remember that the schedule is sorted by time).
Mediator¶
The mediator simply moves tasks in the ready_queue over to the
task pool for execution using
celery.worker.job.TaskRequest.execute_using_pool()
.