KnowledgeHub
Questions
Tags
Users
Search
Alex Rivera
|
Logout
Edit Question
Title
Body
I have 100 servers in my cluster. At time 17:35:00 , all 100 servers are provided with data (of size 1[MB] ). Each server processes the data, and produces an output of about 40[MB] . The processing time for each server is 5[sec] . At time 17:35:05 ( 5[sec] later ), there's a need for a central machine to read all the output from all 100 servers (remember, the total size of data is: 100 [machines] x 40 [MB] ~ 4[GB]), aggregate it, and produce an output. It is of high importance that the entire process of gathering the 4[GB] data from all 100 servers takes as little time as possible . How do I go about solving this problem? Are there any existing tools (ideally, in python , but would consider other solutions) that can help?
Tags (comma-separated)
Save Edits
Cancel