Alex Rivera | Logout

N-gram generation from a sentence

Asked 2010-09-07T07:53:24.603
33

How to generate an n-gram of a string like:

String Input="This is my car."

I want to generate n-gram with this input:

Input Ngram size = 3

Output should be:

This
is
my
car

This is
is my
my car

This is my
is my car

Give some idea in Java, how to implement that or if any library is available for it.

I am trying to use this NGramTokenizer but its giving n-gram's of character sequence and I want n-grams of word sequence.

Edit
Report

1 Answer

1
/**
 * 
 * @param sentence should has at least one string
 * @param maxGramSize should be 1 at least
 * @return set of continuous word n-grams up to maxGramSize from the sentence
 */
public static List<String> generateNgramsUpto(String str, int maxGramSize) {

    List<String> sentence = Arrays.asList(str.split("[\\W+]"));

    List<String> ngrams = new ArrayList<String>();
    int ngramSize = 0;
    StringBuilder sb = null;

    //sentence becomes ngrams
    for (ListIterator<String> it = sentence.listIterator(); it.hasNext();) {
        String word = (String) it.next();

        //1- add the word itself
        sb = new StringBuilder(word);
        ngrams.add(word);
        ngramSize=1;
        it.previous();

        //2- insert prevs of the word and add those too
        while(it.hasPrevious() && ngramSize<maxGramSize){
            sb.insert(0,' ');
            sb.insert(0,it.previous());
            ngrams.add(sb.toString());
            ngramSize++;
        }

        //go back to initial position
        while(ngramSize>0){
            ngramSize--;
            it.next();
        }                   
    }
    return ngrams;
}

Call:

long startTime = System.currentTimeMillis();
ngrams = ToolSet.generateNgramsUpto("This is my car.", 3);
long stopTime = System.currentTimeMillis();
System.out.println("My time = "+(stopTime-startTime)+" ms with ngramsize = "+ngrams.size());
System.out.println(ngrams.toString());

Output:

My time = 1 ms with ngramsize = 9 [This, is, This is, my, is my, This is my, car, my car, is my car]

answered 2012-12-20T17:25:20.163

Your Answer