Alex Rivera | Logout

Django long running asynchronous tasks with threads/processing

Asked 2011-11-09T17:23:47.717
34

Disclaimer: I do know that there are several similar questions on SO. I think I've read most if not all of them, but did not find an answer to my real question (see later). I also do know that using celery or other asynchronous queue systems is the best way to achieve long running tasks - or at least use a cron-managed script. There's also mod_wsgi doc about processes and threads but I'm not sure I got it all correct.

The question is:

what are the exact risks/issues involved with using the solutions listed down there? Is any of them viable for long running tasks (ok, even though celery is better suited)? My question is really more about understanding the internals of wsgi and python/django than finding the best overall solution. Issues with blocking threads, unsafe access to variables, zombie processing, etc.

Let's say:

  1. my "long_process" is doing something really safe. even if it fails i don't care.
  2. python >= 2.6
  3. I'm using mod_wsgi with apache (will anything change with uwsgi or gunicorn?) in daemon mode

mod_wsgi conf:

WSGIDaemonProcess NAME user=www-data group=www-data threads=25
WSGIScriptAlias / /path/to/wsgi.py
WSGIProcessGroup %{ENV:VHOST}

I figured that these are the options available to launch separate processes (meant in a broad sense) to carry on a long running task while returning quickly a response to the user:

os.fork

import os

if os.fork()==0:
    long_process()
else:
    return HttpResponse()

subprocess

import subprocess

p = subprocess.Popen([sys.executable, '/path/to/script.py'], 
                                    stdout=subprocess.PIPE, 
                                    stderr=subprocess.STDOUT)

(where the script is likely to be a manage.py c

django asynchronous mod-wsgi
Edit
Report

1 Answer

30

os.fork

A fork will clone the parent process, which in this case, is your Django stack. Since you're merely wanting to run a separate python script, this seems like an unnecessary amount of bloat.

subprocess

Using subprocess is expected to be interactive. In other words, while you can use this to effectively spawn off a process, it's expected that at some point you'll terminate it when finished. It's possible Python might clean up for you if you leave one running, but my guess would be that this will actually result in a memory leak.

threading

Threads are defined units of logic. They start when their run() method is called, and terminate when the run() method's execution ends. This makes them well suited to creating a branch of logic that will run outside the current scope. However, as you mentioned, they are subject to the Global Interpreter Lock.

multiprocessing

This module allows you to spawn processes, and it has an API similar to that of threading. You could say is like threads on steroids. These processes are not subject to the Global Interpreter Lock, and they can take advantage of multi-core architectures. However, they are more complicated to work with as a result.

So, your choices really come down to threads or processes. If you can get by with a thread and it makes sense for your application, go with a thread. Otherwise, use processes.

answered 2011-11-09T20:06:40.460

Your Answer