8
Most Mac apps are written using Cocoa in Objective-C; which, while it is a compiled language, means that there is a fair bit of information left over that could be used by a decompiler.
I'm not sure if there are a lot of decompilers out there that leverages this information, at least I haven't heard of any.
However, there are also another option; F-Script.
F-Script can be used to attach to an executable and explore its interfaces, while not as good as source, it can give you a pretty clear idea of how the executable is built, and how it operates.
As for the legality issue:
IANAL, but as far as I know, reverse-engineering for the purposes of compatibility is legal in many jurisdictions, and I can't imagine that decompiling an executable to look at its code is illegal, unless the specific EULA specifically prohibits it.
Edit: WRT Steam specifically, it is probably NOT written in Cocoa, but C# with some manner of .NET compatibility layer; and it's probably not a good place to start if you want to learn how to make applications for Mac OS X.
If it's an open-source app, yes. Otherwise it's possible through decompilation but the output will be a real pain in the ass to look at. If you just want the protocols and the interfaces of categories and classes, have a look at class-dump.
I'm not aware of a nib decompiler.
Whether decompilation is legal: ask a lawyer. This may (and probably does) differ per jurisdiction.