This question was originally a homework assignment I had, but my answer was wrong, and I'm curious what is the best solution for this problem.
The goal is to compute key aspects of the "Recommender System bootstrapping algorithm" using 4 map reduce steps. My problem is with the 3rd step, so I'll bring only its details.
input: records of the form:
1. (population id, item, number of rating users, sum of ratings, sum of ratings squared)
2. (population id, splitter item, likers/dislikers, item, number of rating users, sum of ratings, sum of ratings squared)
The 2nd form is pretty much like the 1st form, but a record for each (splitter,likers/dislikers) - where likers/dislikers is a boolean.
This means (I think) there are 2^|items| records of the seconds form for each record from the 1st form... (many classmates made the wrong (again, I think..) assumption that there are the same amount of 1st and 2nd form records)
Task description:
This step will compute, per splitter movie, the squared error (SE) induced by each movie.
- Output: records of the form (population id, splitter item, item, squared error on item given a split on the splitter).
Hint:
assume that there exists a string that precedes (in the system’s sort order) any splitter movie id.
This must be done within one mapreduce step!
additional background:
This was learned at the context of "The Netflix Challange"
SE definition:
EDIT: additional material concerning the problem [some description on the netflix challenge and mathematical information about the problem ] can be found in algorithm mapreduce