I personally think the hyphen as word separator makes for maximum readability and typability in general, so I recommend that where possible (in some contexts, a hyphen can't be used, such as in identifiers in most languages). One important thing to bear in mind is that the scheme you pick will have a bearing on the require statement that users will use with your lib, and you want to avoid having a different gem name than library name.
Bad# gem install my_cool_lib
require 'my-cool-lib'
# gem install MyCoolLib
require 'my_cool_lib'
Good
# gem install my_cool_lib
require 'my_cool_lib'
# gem install my-cool-lib
require 'my-cool-lib'
Unfortunately, a small handful of libraries violate this simple usability rule. Don't be one of those libraries. :)
I would recommend lower case characters with underscores (number 2 in your question). It's true that this naming scheme is the convention in Rails and not necessary in non-Rails projects. However, I would still stick to the Rails convention because most Ruby programmers are probably using Ruby exclusively for Rails anyway.