Celery Exception Handling

Suppose i have this task definition: def some_other_foo(input) raise Exception('This is not handled!') return input @app.task( bind=True, max_retries=5, soft_time_limit=20)

Solution 1:

Try \ except works fine. Your task will always be unsuccessful because you called self.retry before return. Let's make a little test:

from celery import Celery

app = Celery(name_app,broker_settings_etc....)

defsome_other_foo(value):
    raise Exception('This is not handled!')

@app.task(
  bind=True,
  max_retries=5,
  soft_time_limit=20)defsome_foo(self):
    response = ""try:
        response = some_other_foo('test')
    except Exception as exc:
        self.retry(countdown=5, exc=exc)
        response = "error"return response

Run celery app and call our task. You will see in celery logs something like this:

3fb-81de-e4149fa88f4c]retry: Retry in 5s:Exception('Thisisnothandled!',)
[2017-08-18 15:50:34,160:INFO/MainProcess] Received task:tasks.some_foo[b656731b-c85d-43fb-81de-e4149fa88f4c]eta:[2017-08-1812:50:39.156912+00:00]
[2017-08-18 15:50:34,161:INFO/MainProcess] Tasktasks.some_foo[b656731b-c85d-43fb-81de-e4149fa88f4c]retry: Retry in 5s:Exception('Thisisnothandled!',)
[2017-08-18 15:50:39,511:ERROR/MainProcess] Tasktasks.some_foo[b656731b-c85d-43fb-81de-e4149fa88f4c]raised unexpected:Exception('Thisisnothandled!',)Traceback(mostrecentcalllast):# trace here...Exception:Thisisnothandled!

How it works. You set for the task max_retries=5. When you called self.retry(countdown=5, exc=exc) Celery interrupts processing of task and try to restart task with countdown(in our case = 5). After 5 attempts(max_retries) Celery won't rerun task.

Now, let's change our try \ except block to:

try:
    response = some_other_foo('test')
except Exception:
    print'handled'
    response = "bad response"

Restart Celery and run our task. Let's check log:

[2017-08-18 15:58:41,893:INFO/MainProcess] Received task:tasks.some_foo[1437e7ce-1c69-4042-824b-5602f486c025]
[2017-08-18 15:58:41,895:WARNING/Worker-3] handled
[2017-08-18 15:58:41,896:INFO/MainProcess] Tasktasks.some_foo[1437e7ce-1c69-4042-824b-5602f486c025]succeeded in 0.00186271299026s:'bad response'

As you can see handler works fine.

So, summarize. If you call self.retry, Celery will interrupt processing of task and try to restart a current task.