KnowledgeHub
Questions
Tags
Users
Search
Alex Rivera
|
Logout
Edit Question
Title
Body
I understand using subprocess is the preferred way of calling external command. But what if I want to run several commands in parall, but limit the number of processes being spawned? What bothers me is that I can't block the subprocesses. For example, if I call subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile) Then the process will continue, without waiting for cmd to finish. Therefore, I can't wrap it up in a worker of multiprocessing library. For example, if I do: def worker(cmd): subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile); pool = Pool( processes = 10 ); results =[pool.apply_async(worker, [cmd]) for cmd in cmd_list]; ans = [res.get() for res in results]; then each worker will finish and return after spawning a subprocess. So I can't really limit the number of processes generated by subprocess by using Pool . What's the proper way of limiting the number of subprocesses?
Tags (comma-separated)
Save Edits
Cancel