This document describes Celery 3.0. For development docs, go here.
The worker consists of 4 main components: the consumer, the scheduler, the mediator and the task pool. All these components runs in parallel working with two data structures: the ready queue and the ETA schedule.
The ready queue is either an instance of Queue.Queue, or celery.buckets.TaskBucket. The latter if rate limiting is enabled.
Receives messages from the broker using Kombu.
When a message is received it’s converted into a celery.worker.job.TaskRequest object.
Tasks with an ETA are entered into the eta_schedule, messages that can be immediately processed are moved directly to the ready_queue.
The schedule controller is running the eta_schedule. If the scheduled tasks eta has passed it is moved to the ready_queue, otherwise the thread sleeps until the eta is met (remember that the schedule is sorted by time).
The mediator simply moves tasks in the ready_queue over to the task pool for execution using celery.worker.job.TaskRequest.execute_using_pool().