Alex Rivera | Logout

Possibility to apply online algorithms on big data files with sklearn?

Asked 2012-09-17T13:18:56.770
23

I would like to apply fast online dimensionality reduction techniques such as (online/mini-batch) Dictionary Learning on big text corpora. My input data naturally do not fit in the memory (this is why i want to use an online algorithm) so i am looking for an implementation that can iterate over a file rather than loading everything in memory. Is it possible to do this with sklearn ? are there alternatives ?

Thanks register

Edit
Report

1 Answer

4

In addition to Vowpal Wabbit, gensim might be interesting as well - it too features online Latent Dirichlet Allocation.

answered 2012-09-18T11:52:55.733

Your Answer