Alex Rivera | Logout

how to implement eigenvalue calculation with MapReduce/Hadoop?

Asked 2008-12-23T06:29:45.260
10

It is possible because PageRank was a form of eigenvalue and that is why MapReduce introduced. But there seems problems in actual implementation, such as every slave computer have to maintain a copy of the matrix?

Edit
Report

1 Answer

9

PageRank solves the dominant eigenvector problem by iteratively finding the steady-state discrete flow condition of the network.

If NxM matrix A describes the link weight (amount of flow) from node n to node m, then

p_{n+1} = A . p_{n} 

In the limit where p has converged to a steady state (p_n+1 = p_n), this is an eigenvector problem with eigenvalue 1.

The PageRank algorithm doesn't require the matrix to be held in memory, but is inefficient on dense (non-sparse) matrices. For dense matrices, MapReduce is the wrong solution -- you need locality and broad exchange among nodes -- and you should instead look at LaPACK and MPI and friends.

You can see a working pagerank implementation in the wukong library (hadoop streaming for ruby) or in the Heretrix pagerank submodule. (The heretrix code runs independently of Heretrix)

(disclaimer: I am an author of wukong.)

answered 2009-04-21T19:30:47.663

Your Answer