Are there any free parser generators that generate C++ code and handle Unicode correctly?

Asked 2010-11-30T19:42:41.617

After asking this question, I'm now sold on trying to use a parser generator, where before I was going to write things manually.

However, I can't seem to find any such parser that generates C++ code, nor can I find a parser that correctly handles Unicode. (note that my input is in UCS-2 -- I don't care about supporting bits outside of the Basic Multilingual Plane if that makes building the parser more difficult)

There are some parsers which can generate C, but such parsers all seem to throw exception safety out the window, which would prevent me from using C++ inside any semantic actions.

Does a parser generator exist which meets these two tenets, or am I stuck doing everything by hand?

EDIT: Oh, and my project is BSL licensed, so there can't be many restrictions on use of the output of the parser generator itself.

c++ unicode parser-generator

Edit Report

2 Answers

The parser doesn't care about characters since it processes tokens.

Lexing Unicode is very expensive. This is because you either pay a huge function calling overhead for classification, or you kill your memory with massive tables. Normally you'd only support Unicode is specific places in a PL, such as string literals and perhaps identifiers where a handcrafted function can do the job efficiently.

I once coded in Ocamllex a lexer that would accept the identifiers mandated by the ISO C++ standard (which includes a set of ranges of Unicode code points considered as "letters" in various languages). Although the number of code point ranges is quite small (around 20 or so ranges), the UTF-8 DFA for this has over 64K states and blew up the lexer generator :)

My advice here is: you will have to hand craft your lexer. It is, in fact, very easy to do this inefficiently. Doing it efficiently is very much harder: I'd be looking at Judy arrays for support (this is the fastest data structure on the planet).

answered 2010-12-04T20:48:16.873

Try Boost.Spirit. You can plug-in your own "stream decoder", which handles the unicode-part of your problem. To make Sprit work with wchar_t should be possible -- although, I have not tried it myself.

answered 2010-12-09T15:50:38.860

Are there any free parser generators that generate C++ code and handle Unicode correctly?

2 Answers

Your Answer