Alex Rivera | Logout

Limit concurrent Web Service request (Or some batch approach)

Asked 2013-07-07T10:48:00.453
9

I've a for comprehension that fetches a comma separated Id List from a web service.
Then I use the Id List to make new calls, my problem here is that the Id List can be around 10 000 long and each call is a medium sized XML document.
The Web Service end point, or it could be the Play Framework, does't quite like it when I request all the 10 000 at the same time asynchronously as I only get around 500 correct responses.

Some pseudo code to highlight the intent.

for {
  respA <- WS.url(url1).get
  id <- respA.body.split(",")
  respB <- WS.url(url2 + id).get
} yield ...

How do I get about to limit the concurrent request to something more feasible?

web-services scala asynchronous playframework playframework-2.0
Edit
Report

1 Answer

7

You need to do some sort of throttling.

Akka

How about using some Akka Actors to make the requests? Check out these approaches to throttling with akka:

Just with Futures

If you want to just use Futures and no Akka Actors, you could use a combination of flatMap (to chain up HTTP requests to happen one after another) and Future.sequence to get the level of parallelism you want.

answered 2013-07-07T12:33:02.900

Your Answer