66
As mentioned by Jesse Glick in his answer, if you are using Jenkins 1.487 or higher, then there is a native way to uninstall plugins in the Jenkins UI. See JENKINS-3070 for details.
If you are using a version of Jenkins earlier than 1.487, then you can try manually uninstalling the plugin. As some people point out in the comments, this may not work on some platforms (in those cases, upgrade to at least 1.487 so that you can use the official uninstall feature).
To manually uninstall a plugin, stop Hudson/Jenkins, go to your HUDSON_HOME/plugins directory and remove both the .hpi file and the folder with the same name. So, if you were going to remove the CVS plugin, you would remove both the cvs.hpi file and the cvs directory.
After that, restart Hudson/Jenkins and the plugin won't be there anymore.
Hudson Plugins explains that some core plugins ("Tier 1" plugins, as they are called) are shipped with Hudson itself, and I assume cannot be deleted therefore.