Although not documented you can have VCL and Firemonkey in the same application.
There is no compiler define.
If you're building something that needs to be both VCL and Firemonkey I would recommend separation of the units.
A possible way:
Mixing UI code from two different frameworks in the same unit is not a good idea. It will link in the other library when it's not needed.
There is no compiler directive because technically there is no such thing as a firemonkey application or a vcl application. Only applications which make use of these technologies. An application can use fxm or vcl or both or neither (eg. a console app). This is a bit like asking if it is an SQL application. You can of course programatically check the ancestry of individual forms to see which framework they inherit from. Again, inside a unit that has no associated form, this has no meaning.