183
The answer to your question is often, but not always. If your abstract class forbids clients to call delete on a pointer to it (or if it says so in its documentation), you are free to not declare a virtual destructor.
You can forbid clients to call delete on a pointer to it by making its destructor protected. Working like this, it is perfectly safe and reasonable to omit a virtual destructor.
You will eventually end up with no virtual method table, and end up signalling your clients your intention on making it non-deleteable through a pointer to it, so you have indeed reason not to declare it virtual in those cases.
[See item 4 in this article: http://www.gotw.ca/publications/mill18.htm]