Alex Rivera | Logout

Erlang OTP application design

Asked 2011-03-11T22:02:13.667
9

I am struggling a little coming to grips with the OTP development model as I convert some code into an OTP app.

I am essentially making a web crawler and I just don't quite know where to put the code that does the actual work.

I have a supervisor which starts my worker:

-behaviour(supervisor).
-define(CHILD(I, Type), {I, {I, start_link, []}, permanent, 5000, Type, [I]}).

init(_Args) ->          
  Children = [
    ?CHILD(crawler, worker)
  ],  
  RestartStrategy = {one_for_one, 0, 1},
  {ok, {RestartStrategy, Children}}.

In this design, the Crawler Worker is then responsible for doing the actual work:

-behaviour(gen_server).

start_link() ->
  gen_server:start_link(?MODULE, [], []).

init([]) ->
  inets:start(),        
  httpc:set_options([{verbose_mode,true}]), 
  % gen_server:cast(?MODULE, crawl),
  % ok = do_crawl(),
  {ok, #state{}}.

do_crawl() ->
  % crawl!
  ok.

handle_cast(crawl}, State) -> 
  ok = do_crawl(),
  {noreply, State};

do_crawl spawns a fairly large number of processes and requests that handle the work of crawling via http.

Question, ultimately is: where should the actual crawl happen? As can be seen above I have been experimenting with different ways of triggering the actual work, but still missing some concept essential for grokering the way things fit together.

Note: some of the OTP plumbing is left out for brevity - the plumbing is all there and the system all hangs together

Edit
Report

1 Answer

4

Are you actually keeping track of any state in your gen_server?

If the answer is yes, it looks like you are doing things the right way. Note that since messages are serialized, with the above implementation you could not have two crawls going at the same time. If you need concurrent crawls, see the answer to my question here.

If the answer is no then you can possibly ditch the server and the supervisor and just use the application module for any initialization code as seen here.

Finally, lhttpc and ibrowse are considered better alternatives to inets. I use lhttpc in production on my ad servers and it works great.

answered 2011-03-11T23:48:00.687

Your Answer