django, celery, celery worker, celery heart beat 101, broker

Detailed django, celery(worker), redis:

https://testdriven.io/blog/django-and-celery/

detailed django, celery(worker), celery heartbeat:

https://testdriven.io/blog/django-celery-periodic-tasks/

Django communicate to celery through a broker(rabbitmq, redis db),

Django send tasks to broker, then celery picks up tasks there.

For Django to send tasks to broker, it needs to have celery installed and use its function.

For celery to pick up tasks celery needs to spawn a woker, celery should only spawn 1 worker, 1 worker can create many child process it depends on how many cpu core you have.

https://medium.com/@iamlal/scale-up-messaging-queue-with-python-celery-processes-vs-threads-402533be269e#:~:text=Celery%20recommends%201%20worker%20per,that%20number%20of%20CPU%20cores.

You can have the above django container to spawn worker, as well but if worker fail you dont want the django application to fail, since separating it and use same volume is better

!!!

https://stackoverflow.com/questions/75245127/why-would-you-separate-a-celery-worker-and-django-container

https://stackoverflow.com/questions/36439024/can-you-run-celery-in-a-different-container-from-django

...
python manage.py migrate
celery -A api worker -l INFO --detach
python manage.py runserver 0.0.0.0:8000

!!!!

worker sampler:

https://www.revsys.com/tidbits/celery-and-django-and-docker-oh-my/

celery:

    build: .

    command: celery -A proj worker -l info

    volumes:

      - .:/code

    depends_on:

      - db

      - redis

This code adds a Celery worker to the list of services defined in docker-compose. Now our app can recognize and execute tasks automatically from inside the Docker container once we start Docker using docker-compose up.

The celery worker command starts an instance of the celery worker, which executes your tasks. -A proj passes in the name of your project, proj, as the app that Celery will run. -l info

Celery heart beat is to schedule jobs to celery worker, 

https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html

need to sepcify timezone(its using crontab)

You can have the above django container to spawn heart beat, as well but if worker fail you dont want the django application to fail, since separating it and use same volume is better

MSG broker for celery and django can be redis or RabbitMQ

https://docs.celeryq.dev/en/latest/getting-started/first-steps-with-celery.html#keeping-results

Keeping Results

If you want to keep track of the tasks’ states, Celery needs to store or send the states somewhere. There are several built-in result backends to choose from: SQLAlchemy/Django ORM, MongoDB, Memcached, Redis, RPC (RabbitMQ/AMQP), and – or you can define your own.

rabbit MQ sampler

pp = Celery('test_celery',

             broker='amqp://jimmy:jimmy123@localhost/jimmy_vhost',

             backend='rpc://',

             include=['test_celery.tasks'])

Here, we initialize an instance of Celery called app, which is used later for creating a task.

The first argument of Celery is just the name of the project package, which is “test_celery”.

The broker argument specifies the broker URL, which should be the RabbitMQ we started earlier. Note that the format of broker URL should be:

transport://userid:password@hostname:port/virtual_host

https://tests4geeks.com/blog/python-celery-rabbitmq-tutorial/#:~:text=For%20RabbitMQ%2C%20the%20transport%20is,set%20a%20backend%20for%20Celery.

For RabbitMQ, the transport is amqp.

The backend argument specifies a backend URL. A backend in Celery is used for storing the task results. So if you need to access the results of your task when it is finished, you should set a backend for Celery.

rpc means sending the results back as AMQP messages, which is an acceptable format for our demo. More choices for message formats can be found here.

rabbitMQ git sampler:

https://gist.github.com/mmautner/b0821fa054cf584db6275f6253e740ca

Django / celery mult thread

 django async :

https://docs.djangoproject.com/en/4.1/topics/async/

python thread library:

https://docs.python.org/3/library/threading.html

down side for python threading for web app is that 1 user sends a request, multi threads run, server has to wait for desired thread to finish to give response.

Task based threading :

http request send, server immidately responbds. multi thread jobs are then ran in the background.

Faster response, more complex setup, acheived through celery:

https://medium.com/@ravisarath64/are-you-working-on-django-is-celery-confuse-you-629fedf8287b

Working on technologies like celery can be confusing at first. But it is not as hard as you think. I will briefly explain Celery and its uses while working with Django in this article.

Celery is a nice tool to use whenever you don’t want to wait for some other process to finish. Let us see this with an example.

in the normal way

User sends request
Django receives => spawns a thread to do something else.
wait for the main thread to finishes & the other thread to finishes after completion of both tasks
response is sent to the user as a package                          😭 --> it takes too much time.

by using celery

User sends request
Django receives => lets Celery know "hey! do this!"
main thread finishes
response is sent to the user
The user receives the balance of the transaction                           😄 --> it's so fast

Can Django do multi-thread works? Yes it can do multi-thread works, but generally one uses Celery to do the equivalent

Wait a minute what is single-threaded & multi-threaded?

Single-threaded processes contain the execution of instructions in a single sequence whereas, multithreaded processes allow the execution of multiple parts of a program at the same time.

let's look into deep.

Celery is an asynchronous queue/job based on distributed message passing. For managing the queue, Celery requires a solution to send and receive messages; usually, this comes in the form of a separate service called a message broker.

Redis, RabbitMQ, Amazon SQS are the choices available for brokers.

With Asynchronous execution, you will begin a task and while running the task in the background you can start your next task, then at some point, say “wait for this to finish”. Let’s look at an example:

Django says => Celery starts a task A.Django says => Celery starts a task B.Django says => Celery starts a task C.at some point Celery says =>  Wait for A to finish

The advantage is that you can execute both tasks B & C while A is still running in the background, on a separate thread. This allows you to take better advantage of your resources.

Is celery used only for this?

No, it supports scheduling as well. Let us see this with an example.

User sends request
Django receives => Celery send an email at midnight for the user.
main thread finishesthe response is sent to the user

Celery sends the email at midnight. This is how task scheduling is done in celery.

Let us see another example.

User sends request
Django receives => Celery sent alert to the user in every 10 min.
main thread finishesthe response is sent to the user

celery will send an alert to the user every 10 mins. so celery can be used to schedule as well as to do the periodic task.

Let’s see how to install celery.

You can install celery using standard Python tools like pip or easy_install:

pip install celery

Now connect the celery to your application

First, you have to add the broker URL in project settings.py. You can use either Redis or RabbitMQ. Here I am using Redis

BROKER_URL = 'redis://localhost:6379'

Add a new file called celery.py inside your Django project

from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')
app = Celery('picha')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)# If you need to schedule a task or run a task periodically  you can # add this code app.conf.beat_schedule = {                                                               
   # to run a task daily at 12.30 AM midnight                                'generate_daily_settlement_report’: {
        'task’: 'app.tasks.function_A’,
        'schedule’: crontab(hour=1, minute=30),
 },
   # to run a task in every 30 seconds
 'add-every-30-seconds’: {
        'task’: 'app.tasks.function_B’,
        'schedule’: 30.0,
        'args’: (16, 16)
  },
}@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request)) 

Next, we will create a file tasks.py in your app. Then create the functions, function_A, function_B, etc in your app.

from celery.decorators import task

@task(name="function A")
def function_A():
    '''do something '''@task(name="function B")
def function_B():
    '''do something '''@task(name="function C")
def function_C():
    '''do something '''

let the function_C another celery task to do something else. This function can be called from our view using the delay() method. So we can do the task in other threads without affecting the main thread

function_C.delay()

What’s next?

we have to run the celery worker server

let’s run the celery worker server

You can now run your worker by executing the program with the worker argument:

celery -A tasks worker --loglevel=INFO

so the worker will execute the task. we have another argument called beat which will kick the periodic as well as scheduled tasks.

celery -A tasks beat --loglevel=INFO

celery beat is a scheduler; It kicks off tasks at regular intervals, that are then executed by available worker nodes in the cluster. By default, the entries are taken from the beat_schedule setting, but custom stores can also be used, like storing the entries in a SQL database.

Celery beat is a scheduler. It kicks off tasks at regular intervals, which are then executed by available worker nodes in the cluster. By default, the entries are taken from the beat_schedule setting, but custom stores can also be used, like storing the entries in an SQL database.

A short version of the steps you need to do:

• Run your Django and Redis.
• Open two new terminal windows/tabs.
• In each new window, navigate to your project directory.
• Activate your virtualenv.
• Run the celery beat and worker commands.

Conclusion

This stuffs are only for beginners for more details you use the below links.

redis default pwd, default port, default connection to celery

celery connection to redis in python or  django:

 https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html#configuration

https://stackoverflow.com/questions/68090167/how-to-connect-celery-with-redis

in django set up the following in settings.py:

CELERY_BROKER_URL='redis://localhost:6379',
CELERY_RESULT_BACKEND='redis://localhost:6379'

These two entries give your Celery application instance 

enough information to know where to send messages and where to record the 

results of scheduled jobs

CELERY_BROKER_URL = send message

CELERY_RESULT_BACKEND = record results

in python:

app.conf.broker_url = 'redis://localhost:6379/0'

the url format is :

redis://:password@hostname:port/db_number

all fields after the scheme are optional, and will default to localhost on port 6379, using database 0.

redis (cache db) :

default tcp port 6379

start with a different port : 

https://stackoverflow.com/questions/27895165/how-to-start-redis-server-on-a-different-port-than-the-default-port-6379-in-ubun

redis-server --port 6380

default database : 0

https://www.digitalocean.com/community/cheatsheets/how-to-manage-redis-databases-and-keys

Redis databases are numbered from 0 to 15 and, by default, you connect to database 0 when you connect to your Redis instance. 

to change :

select 15

default pwd: redis requires no pwd by default 

to enable pwd :

https://stackoverflow.com/questions/7537905/how-to-set-password-for-redis

01) open redis configuration file

sudo vi /etc/redis/redis.conf

find requirepass field under SECURITY section and uncomment that field.Then set your password instead of "foobared"

# requirepass foobared

It should be like,

requirepass YOUR_PASSWORD

Then restart redis and start redis-cli.

If you need to check whether you have set the password correctly, you can run below commads in redis-cli.

sithara@sithara-X555UJ ~ $ redis-cli

127.0.0.1:6379> set key1 18

(error) NOAUTH Authentication required.

127.0.0.1:6379> auth admin

OK

127.0.0.1:6379> get key1

(nil)

127.0.0.1:6379> exit

sithara@sithara-X555UJ ~ $ redis-cli

127.0.0.1:6379> set key1 18

(error) NOAUTH Authentication required.

127.0.0.1:6379> auth admin

OK

127.0.0.1:6379> set key2 check

OK

127.0.0.1:6379> get key2

"check"

127.0.0.1:6379> get key1

(nil)

127.0.0.1:6379> set key1 20

OK

127.0.0.1:6379> get key1

"20"

127.0.0.1:6379> exit