26
You could also use LinqPad, as that doesn't involve an install of any kind - http://www.linqpad.net/.
Run the following code as a "C# Statement(s)":
System.Diagnostics.PerformanceCounterCategory.Delete("Name of category to delete");
I'd run it twice, first time to do the actual delete, second time to get an error message to confirm the delete was successful.