What should a "unit" be when unit testing?

It may seem trivial to quote Wikipedia, but I think it's very succinct and accurate in this case:

A unit is the smallest testable part of an application.

This seems to agree with the comment in your question that a unit is "the smallest piece of the code that can be usefully tested". In other words, make the unit as small as you possibly can such that it still makes sense to the developer/tester by itself.

Often you will want to test parts of a project in isolation, and then test how they interact in combination. Having various tiers (levels) of unit testing is often a wise thing to do, as it helps insure that your code is working as it should on all levels, from individual functions up to entire self-contained tasks. I personally do not believe that it is wrong, or even unhelpful, to test individual functions, so long as they are doing something useful in themselves, which can often be the case.

To be quite honest, there is no definite or rigorous definition of a "unit" in "unit testing", which is precisely why the vague term "unit" is used! Learning what needs to be tested and at what level is a matter of experience, and quite often, simply trial and error. It may sound like a slightly unsatisfying answer, but I believe it is a sound rule to follow.

I've always tested at the function level, and that's worked just fine. The big point for me is that unit testing tests the contract between two pieces of code. The unit test just acts as a caller, and ensures that the code being tested (no matter how large - a single function or a huge, nested library that takes 30 minutes to test) returns results in a predictable way.

The whole point of a unit test it to ensure compatibility (by not breaking intereactions) and ensure predictable results. Testing at any place there's an exchange of information will help stabilize your application.

UPDATE: I've also always added a unit test whenever a customer or user reports an edge case that breaks an interaction in my application. Fixing it isn't sufficient for me - adding a unit test to ensure that the fix holds prevents regression, and helps keep things stable down the line.