I'm trying to interface a large Scala + Akka + PlayMini application with an external REST API. The idea is to periodically poll (basically every 1 to 10 minutes) a root URL and then crawl through sub-level URLs to extract data which is then sent to a message queue.
I have come up with two ways to do this:
1st way
Create a hierarchy of actors to match the resource path structure of the API. In the Google Latitude case, that would mean, e.g.
- Actor 'latitude/v1/currentLocation' polls https://www.googleapis.com/latitude/v1/currentLocation
- Actor 'latitude/v1/location' polls https://www.googleapis.com/latitude/v1/location
- Actor 'latitude/v1/location/1' polls https://www.googleapis.com/latitude/v1/location/1
- Actor 'latitude/v1/location/2' polls https://www.googleapis.com/latitude/v1/location/2
- Actor 'latitude/v1/location/3' polls https://www.googleapis.com/latitude/v1/location/3
- etc.
In this case, each actor is responsible for polling its associated resource periodically, as well as creating / deleting child actors for next-level path resources (i.e. actor 'latitude/v1/location' creates actors 1, 2, 3, etc. for all locations it learns about through polling of https://www.googleapis.com/latitude/v1/location).
2nd way
Create a pool of identical polling actors which receive polling requests (containing the resource path) load-balanced by a router, poll the URL once, do some processing,