Edit Question - KnowledgeHub

Title Body

Im looking to build a distributed video encoding cluster of a few dozen machines. Ive never worked with a messaging queue before, but the 2 that I started playing around with were Gearman and Beanstalkd. Beanstalk seems to be a lot simpler and easier to use than Gearman, but its not as feature rich as. One thing I don't understand is... how do you spawn new workers on all the servers? I plan to use php. Is it as simple as running worker.php in CLI with "&" and just have it sit there waiting for work? I noticed gearman doesn't actually kill the process after a job is done, but Beanstalk does, so I have to restart the script after every job, on every server. Currently Im more inclined to use Beanstalk, the general flow of things I planned was: Run a minutely cron on each server that checks if there are pre-defined amount of workers running. If its less than supposed to be, spawn new worker processes. Each process will take roughly 2-30 minutes. Maybe I have a flaw in my logic here? Let me know what would be a "better" or "proper" way of doing this?

Tags (comma-separated) Cancel