Alex Rivera | Logout

Detect if a process is already running and collaborate with it

Asked 2010-01-08T22:12:53.680
9

I'm trying to create a program that starts a process pool of, say, 5 processes, performs some operation, and then quits, but leaves the 5 processes open. Later the user can run the program again, and instead of it starting new processes it uses the existing 5. Basically it's a producer-consumer model where:

  1. The number of producers varies.
  2. The number of consumers is constant.
  3. The producers can be started at different times by different programs or even different users.

I'm using the builtin multiprocessing module, currently in Python 2.6.4., but with the intent to move to 3.1.1 eventually.

Here's a basic usage scenario:

  1. Beginning state - no processes running.
  2. User starts program.py operation - one producer, five consumers running.
  3. Operation completes - five consumers running.
  4. User starts program.py operation - one producer, five consumers running.
  5. User starts program.py operation - two producers, five consumers running.
  6. Operation completes - one producer, five consumers running.
  7. Operation completes - five consumers running.
  8. User starts program.py stop and it completes - no processes running.
  9. User starts program.py start and it completes - five consumers running.
  10. User starts program.py operation - one procucer, five consumers running.
  11. Operation completes - five consumers running.
  12. User starts program.py stop and it completes - no processes running.

The problem I have is that I don't know where to start on:

  1. Detecting that the consumer processes are running.
  2. Gaining access to them from a previously unrelated program.
  3. Doing 1 and 2 in a cross-platform way.

Once I can do that, I know how to manage the processes. There has to be some reliab

python multiprocessing
Edit
Report

1 Answer

1

You need a client-server model on a local system. You could do this using TCP/IP sockets to communicate between your clients and servers, but it's faster to use local named pipes if you don't have the need to communicate over a network.

The basic requirements for you if I understood correctly are these:
1. A producer should be able to spawn consumers if none exist already.
2. A producer should be able to communicate with consumers.
3. A producer should be able to find pre-existing consumers and communicate with them.
4. Even if a producer completes, consumers should continue running.
5. More than one producer should be able to communicate with the consumers.

Let's tackle each one of these one by one:

(1) is a simple process-creation problem, except that consumer (child) processes should continue running, even if the producer (parent) exits. See (4) below.

(2) A producer can communicate with consumers using named pipes. See os.mkfifo() and unix man page of mkfifo() to create named pipes.

(3) You need to create named pipes from the consumer processes in a well known path, when they start running. The producer can find out if any consumers are running by looking for this well-known pipe(s) in the same location. If the pipe(s) do not exist, no consumers are running, and the producers can spawn these.

(4) You'll need to use os.setuid() for this, and make the consumer processes act like a daemon. See unix

answered 2010-01-09T03:50:15.563

Your Answer