Edit Question - KnowledgeHub

Title Body

I'm trying to do Naive Bayes on a dataset that has over 6,000,000 entries and each entry 150k features. I've tried to implement the code from the following link: Implementing Bag-of-Words Naive-Bayes classifier in NLTK The problem is (as I understand), that when I try to run the train-method with a dok_matrix as it's parameter, it cannot find iterkeys (I've paired the rows with OrderedDict as labels): Traceback (most recent call last): File "skitest.py", line 96, in <module> classif.train(add_label(matr, labels)) File "/usr/lib/pymodules/python2.6/nltk/classify/scikitlearn.py", line 92, in train for f in fs.iterkeys(): File "/usr/lib/python2.6/dist-packages/scipy/sparse/csr.py", line 88, in __getattr__ return _cs_matrix.__getattr__(self, attr) File "/usr/lib/python2.6/dist-packages/scipy/sparse/base.py", line 429, in __getattr__ raise AttributeError, attr + " not found" AttributeError: iterkeys not found My question is, is there a way to either avoid using a sparse matrix by teaching the classifier entry by entry (online), or is there a sparse matrix format I could use in this case efficiently instead of dok_matrix? Or am I missing something obvious? Thanks for anyone's time. :) EDIT, 6th sep: Found the iterkeys, so atleast the code runs. It's still too slow, as it has taken several hours with a dataset of the size of 32k, and still hasn't finished. Here's what I got at the moment: matr = dok_matrix((6000000, 150000), dtype=float32) labels = OrderedDict() #collect the data into the matrix pipeline = Pipeline([('nb', MultinomialNB())]) classif = SklearnClassifier(pipeline) add_label = lambda lst, lab: [(lst.getrow(x).todok(), lab[x]) for x in xrange(lentweets-foldsize)] classif.train(add_label(matr[:(lentw

Tags (comma-separated) Cancel