I'm writing a lexer for the IMAP protocol for educational purposes and I'm stumped as to where I should draw the line between lexer and parser. Take this example of an IMAP server response:
* FLAGS (\Answered \Deleted)
This response is defined in the formal syntax like this:
mailbox-data = "FLAGS" SP flag-list
flag-list = "(" [flag *(SP flag)] ")"
flag = "\Answered" / "\Deleted"
Since they are specified as string literals (aka "terminal" tokens) would it be more correct for the lexer to emit a unique token for each, like:
(TknAnsweredFlag)
(TknSpace)
(TknDeletedFlag)
Or would it be just as correct to emit something like this:
(TknBackSlash)
(TknString "Answered")
(TknSpace)
(TknBackSlash)
(TknString "Deleted")
My confusion is that the former method could overcomplicate the lexer - if \Answered had two meanings in two different contexts the lexer wouldn't emit the right token. As a contrived example (this situation won't occur because e-mail addresses are enclosed in quotes), how would the lexer deal with an e-mail address like \Answered@googlemail.com? Or is the formal syntax designed to never allow such an ambiguity to arise?