How Can Celery Distribute Users' Tasks In A Fair Way?

The task I'm implementing is related to scrape some basic info about a URL, such as title, description and OGP metadata. If User A requests 200 URLs to scrape, and after User B req

Solution 1:

Another way would be to rate limit individual users using a lock. Use the user id as the lock name. If the lock is already held retry after some task dependent delay.

Basically, do this:

Ensuring a task is only executed one at a time

Lock on the user id and retry instead of doing nothing if the lock can't be acquired. Also, it would be better to use Redis instead of the the Django cache, but either way will work.

Solution 2:

One way to work this around could be to control that a user does not enqueue more than x tasks, which means counting for each user the number of non-processed tasks enqueued (on the django side, not trying to do this with celery).

Solution 3:

How about, instead of running all URL scrapes in a single task, make each scrape into a single task and then run them as chains or groups?