Alex Rivera | Logout

Clean text coming from PDFs

Asked 2012-05-02T14:42:31.503
9

this is more of an algorithmic question rather than a specific language question, so I am happy to receive an answer in any language - even pseudocode, even just an idea.

Here is my problem: I need to work on large dataset of papers that come from articles in PDF and that were brutally copied/pasted into .txt. I only have the result of this abomination, which is around 16k papers, for 3.5 GB or text (the corpus I am using is the ACL Antology Network, http://clair.si.umich.edu/clair/aan/DatasetContents.html ).

The "junk" comes from things like formulae, images, tables, and so on. It just pops in the middle of the running text, so I can't use regular expressions to clean it, and I can't think of any way to use machine learning for it either. I already spent a week on it, and then I decided to move on with a quick&dirty fix. I don't care about cleaning it completely anymore, I don't care about false negatives and positives as long as the majority of this areas of text is removed.

Some examples of the text: note that formulae contain junk characters, but tables and caption don't (but they still make my sentence very long, and thus unparsable). Junk in bold.

Easy one:

The experiments were repeated while inhibiting specialization of first the scheme with the most expansions, and then the two most expanded schemata. Measures of coverage and speedup are important 1 As long as we are interested in preserving the f-structure assigned to sentences, this notion of coverage is stricter than necessary. The same f-structure can in fact be assigned by more than one parse, so that in some cases a sentence is considered out of coverage even if the specialized grammar assigns to it the correct f-structure. 2'VPv' and 'VPverb[main]' cover VPs headed by a main verb. 'NPadj' covers NPs with adjectives attached. 205 The original rule: l/Ppe

language-agnostic nlp stanford-nlp
Edit
Report

1 Answer

2

(I hate crappy copy&pastes. )

Few ideas that you might find helpful (I used each and every one of them myself in that point or another)

  1. (Very brute force) : Using a tokenizer and a dictionary (real dictionary, not the data structure) - parse the words out and any word which is not a dictionary word - remove it. It might prove problematic if your text contains a lot of company/products names - but this too can be solved using the correct indexes (there are a few on the web - I'm using some propriety ones so I can't share them, sorry)

  2. Given a set of clean documents (lets say a 2K), build an tf/idf index of them, and use this as a dictionary - every term from the other documents that doesn't appear in the index (or appears with a very low tf/idf) - remove it. This should give you a rather clean document.

  3. Use Amazon's mechanical turk mechanism : set up a task where the person reading the document needs to mark the paragraph that doesn't make sense. Should be rather easy for the mechanical turk platform (16.5K is not that much) - this will probably cost you a couple of hundred $ , but you'll probably get a rather nice cleanup of the text (So if it's on corporate money, that can be your way out - they need to pay for their mistakes :) ).

  4. Considering your documents are from the same domain (same topics, all in all), and the problems are quite the same (same table headers, roughly same formulas): Break all the documents to sentences, and try clustering the sentences using ML. If the table headers / formulas are relatively similar, they should cluster nicely away from the rest of the sentences, and then you can clean the documents sentence-by-sentence (Get a document, break it to sentences, for each sentence, if it's part of the "weird" cluster, remove it)

answered 2012-05-14T15:47:34.057

Your Answer