This document describes an older version of Celery (2.5). For the latest stable version please go here.
The worker consists of 4 main components: the consumer, the scheduler, the mediator and the task pool. All these components runs in parallel working with two data structures: the ready queue and the ETA schedule.
The ready queue is either an instance of Queue.Queue, or celery.buckets.TaskBucket. The latter if rate limiting is enabled.
The ETA schedule is a heap queue sorted by time.
Receives messages from the broker using Kombu.
When a message is received it’s converted into a celery.worker.job.TaskRequest object.
Tasks with an ETA are entered into the eta_schedule, messages that can be immediately processed are moved directly to the ready_queue.
The schedule controller is running the eta_schedule. If the scheduled tasks eta has passed it is moved to the ready_queue, otherwise the thread sleeps until the eta is met (remember that the schedule is sorted by time).