Untangled Development

Do you need a task queue in your web app?

Within your web application, some operations, or “tasks”, take an unpredictable time to run.

Others take too long to have them executed within the web application’s request/response cycle.

Adding a task queue will allow you to:

  • Run longer-running tasks outside the request-response cycle handled by web framework. Actually gunicorn or uwsgi in a production Django1 setup.
  • Run scheduled tasks by not relying on crontab. This usually entails management commands or other forms of standalone scripts. Each needing all environment variables loaded correctly.

Without a task queue

Let’s say you want to send an email to a user when they submit a form.

The process is shown below2:

Figure 1. View sends email directly. No queue.

Figure 1. View sends email directly. No queue.

This adds an unpredictable time to the expected response time. I.e. those 1-2 seconds taken to:

  • talk to the email provider, over the network
  • receive a response from the email provider, again over the network

With a task queue

With the task queue in place, the email sending part is done asynchronously. I.e. The unpredictable part is shifted onto the queue. And we don’t really care how long that takes; that is, for returing a response to the user.

In case it fails, many task queues offer “task retrying”. I.e. the task would automatically be retried for a pre-set number of times.

This does not only result in a speed improvement. But in a better architecture. The part which could be “done later”, is now delegated to the task queue:

Figure 2. View dispatches send_email task onto queue.

Figure 2. View dispatches send_email task onto queue.

Notice, that for the sake of examples, the queue is handling other tasks, such as scheduled tasks.

Let’s consider a particular case with a long-running process. Such as that to generate a CSV file. Which takes longer than 60 seconds.

This wouldn’t even be possible in the response/request cycle. Because most Web servers, such as Nginx, time out after 60 seconds by default.

A task queue allows our web app user to trigger (“enqueue”) a longer-running operation while keeping the application responsive!

Notice how these tasks can be queued from various places callers, not just from the request/response cycle.

Conclusion

There are several task queues in the Python world.

Going through each and everyone of them is not the objective of this article.

But I did write up a little guide on having a minimal queue setup with Huey and Django. Let me know what you think!

Footnotes

  1. Django is used as example web application framework for this post. Still most of the arguments are applicable to any backend web framework, as Flask or Rails. ↩

  2. Diagrams above drawn using the excellent excalidraw.com! ↩

Comments !