12
I wrote a TokenFilter which adds tokens in a stream.
1. Tests shows it works, but I don't completely understand why.
If someone could shed a light on the semantics I'd be grateful. In particular, at (*), restoring the state, doesn't that mean we either overwrite the current token or the token created before capturing the state?
This is roughly what I did
private final LinkedList<String> extraTokens = new LinkedList<String>();
private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
private State savedState;
@Override
public boolean incrementToken() throws IOException {
if (!extraTokens.isEmpty()) {
// Do we not loose/overwrite the current termAtt token here? (*)
restoreState(savedState);
termAtt.setEmpty().append(extraTokens.remove());
return true;
}
if (input.incrementToken()) {
if (/* condition */) {
extraTokens.add("fo");
savedState = captureState();
}
return true;
}
return false;
}
Does that mean, for an input stream of whitespace tokenized string "a b c"
(a) -> (b) -> (c) -> ...
where bb is a new synonym to b, that the graph will be constructed like this when restoreState is used?
(a)
/ \
(b) (bb)
\ /
(c)
|
...
2. Attributes
Given the text foo bar baz with fo being the stem of foo and qux being synonym to bar baz, have I constructed the correct attribute table?
+--------+---------------+-----------+--------------+-----------+
| Term | startOffset | endOffset | posIncrement | posLenght |
+--------+---------------+-----------+--------------+-----------+
| foo | 0 | 3 |