I wrote a TokenFilter which adds tokens in a stream.

1. Tests shows it works, but I don't completely understand why.

If someone could shed a light on the semantics I'd be grateful. In particular, at (*), restoring the state, doesn't that mean we either overwrite the current token or the token created before capturing the state?

This is roughly what I did

private final LinkedList<String> extraTokens = new LinkedList<String>();
private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
private State savedState;

@Override
public boolean incrementToken() throws IOException {
    if (!extraTokens.isEmpty()) {
        // Do we not loose/overwrite the current termAtt token here? (*)
        restoreState(savedState);
        termAtt.setEmpty().append(extraTokens.remove());
        return true;
    }
    if (input.incrementToken()) {
        if (/* condition */) {
           extraTokens.add("fo");
           savedState = captureState();
        }
        return true;
    }
    return false;
}

Does that mean, for an input stream of whitespace tokenized string "a b c"

 (a) -> (b) -> (c) -> ...

where bb is a new synonym to b, that the graph will be constructed like this when restoreState is used?

    (a)
   /   \
(b)    (bb)
   \   /
    (c)
     |
    ...

2. Attributes

Given the text foo bar baz with fo being the stem of foo and qux being synonym to bar baz, have I constructed the correct attribute table?

+--------+---------------+-----------+--------------+-----------+
|  Term  |  startOffset  | endOffset | posIncrement | posLenght |
+--------+---------------+-----------+--------------+-----------+
|  foo   |       0       |     3     | 
Edit
Report