Why should you remove unnecessary C# using directives?

For example, I rarely need:

using System.Text;

but it's always there by default. I assume the application will use more memory if your code contains unnecessary using directives. But is there anything else I should be aware of?

Also, does it make any difference whatsoever if the same using directive is used in only one file vs. most/all files?

Edit: Note that this question is not about the unrelated concept called a using statement, designed to help one manage resources by ensuring that when an object goes out of scope, its IDisposable.Dispose method is called. See Uses of "using" in C#.