Entity Framework AsNoTracking breaks call to Distinct

I am trying to load a list of distinct colors from previously loaded list of products on a page. So to pull in the products I do this:

var products = Products
    .Include(p => p.ProductColor)
    .ToList();

Then I do some processing on the products them I want to get a list of all of the distinct colors used by the products, so I do this:

var colors = products   
    .Select(p => p.ProductColor)
    .Distinct();

And this works great, however if I add a call to .AsNoTracking() to the original products call, I now get an entry in my color list for each entry in the product list.

Why is there a difference in these two? Is there a way to keep Entity Framework from tracking the objects (they're being used for read only) and to get the desired behavior?

Here is my query after adding the call to AsNoTracking()

var products = Products
    .AsNoTracking()
    .Include(p => p.ProductColor)
    .ToList();