KnowledgeHub
Questions
Tags
Users
Search
Alex Rivera
|
Logout
Edit Question
Title
Body
Can I use a Celery Group primitive as the umbrella task in a map/reduce workflow? Or more specific: Can the subtasks in a Group be run on multiple workers on multiple servers ? From the docs: However, if you call apply_async on the group it will send a special grouping task, so that the action of calling the tasks happens in a worker instead of the current process That seems to imply the tasks are all send to one worker... Before 3.0 (and still) one could fire off the subtasks in a TaskSet which would run on multiple servers. The problem is determining whether all tasks have finished executing. That is normally done by polling all subtasks which is not really elegant. I am wondering if the Group primitive can be used to mitigate this problem.
Tags (comma-separated)
Save Edits
Cancel