Edit Question - KnowledgeHub

Title Body

I'm writing a lexer for the IMAP protocol for educational purposes and I'm stumped as to where I should draw the line between lexer and parser. Take this example of an IMAP server response: * FLAGS (\Answered \Deleted) This response is defined in the formal syntax like this: mailbox-data = "FLAGS" SP flag-list flag-list = "(" [flag *(SP flag)] ")" flag = "\Answered" / "\Deleted" Since they are specified as string literals (aka "terminal" tokens) would it be more correct for the lexer to emit a unique token for each, like: (TknAnsweredFlag) (TknSpace) (TknDeletedFlag) Or would it be just as correct to emit something like this: (TknBackSlash) (TknString "Answered") (TknSpace) (TknBackSlash) (TknString "Deleted") My confusion is that the former method could overcomplicate the lexer - if \Answered had two meanings in two different contexts the lexer wouldn't emit the right token. As a contrived example (this situation won't occur because e-mail addresses are enclosed in quotes), how would the lexer deal with an e-mail address like \Answered@googlemail.com? Or is the formal syntax designed to never allow such an ambiguity to arise?

Tags (comma-separated) Cancel