Edit Question - KnowledgeHub

Title Body

Problem: I would like to download 100 files in parallel from AWS S3 using their .NET SDK. The downloaded content should be stored in 100 memory streams (the files are small enough, and I can take it from there). I am geting confused between Task, IAsyncResult, Parallel.*, and other different approaches in .NET 4.0. If I try to solve the problem myself , off the top of my head I imagine something like this pseudocode: (edited to add types to some variables) using Amazon; using Amazon.S3; using Amazon.S3.Model; AmazonS3 _s3 = ...; IEnumerable<GetObjectRequest> requestObjects = ...; // Prepare to launch requests var asyncRequests = from rq in requestObjects select _s3.BeginGetObject(rq,null,null); // Launch requests var asyncRequestsLaunched = asyncRequests.ToList(); // Prepare to finish requests var responses = from rq in asyncRequestsLaunched select _s3.EndGetRequest(rq); // Finish requests var actualResponses = responses.ToList(); // Fetch data var data = actualResponses.Select(rp => { var ms = new MemoryStream(); rp.ResponseStream.CopyTo(ms); return ms; }); This code launches 100 requests in parallel, which is good. However, there are two problems: The last statement will download files serially, not in parallel. There doesn't seem to be BeginCopyTo()/EndCopyTo() method on stream... The preceding statement will not let go until all requests have responded. In other words none of the files will start downloading until all of them start. So here I start thinking I am heading down the wrong path... Help?

Tags (comma-separated) Cancel