Edit Question - KnowledgeHub

Title Body

I wrote a TokenFilter which adds tokens in a stream. 1. Tests shows it works, but I don't completely understand why. If someone could shed a light on the semantics I'd be grateful. In particular, at (*) , restoring the state, doesn't that mean we either overwrite the current token or the token created before capturing the state? This is roughly what I did private final LinkedList<String> extraTokens = new LinkedList<String>(); private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class); private State savedState; @Override public boolean incrementToken() throws IOException { if (!extraTokens.isEmpty()) { // Do we not loose/overwrite the current termAtt token here? (*) restoreState(savedState); termAtt.setEmpty().append(extraTokens.remove()); return true; } if (input.incrementToken()) { if (/* condition */) { extraTokens.add("fo"); savedState = captureState(); } return true; } return false; } Does that mean, for an input stream of whitespace tokenized string "a b c" (a) -> (b) -> (c) -> ... where bb is a new synonym to b , that the graph will be constructed like this when restoreState is used? (a) / \ (b) (bb) \ / (c) | ... 2. Attributes Given the text foo bar baz with fo being the stem of foo and qux being synonym to bar baz , have I constructed the correct attribute table? +--------+---------------+-----------+--------------+-----------+ | Term | startOffset | endOffset | posIncrement | posLenght | +--------+---------------+-----------+--------------+-----------+ | foo | 0 | 3 |

Tags (comma-separated) Cancel