Help me understand "LINQ to Entities only supports casting Entity Data Model primitive types"

I have a unit of work and a repository using EF 4 and POCOs. Since EF requires an ordered set before it can Skip() and Take(), I added the following unit test (without mocks) just to pull out a record to see if it worked.

var myList = UOW.EntityRepo.Get( orderbyLambda: p => p.ID, page: 1, pageSize: 1);

This results in an expression of orderbyLambda = {p => Convert(p.ID)} and an error during enumeration. The ID is a tinyint (Int16 / short)

So why does this fail to order by the ID? More about the error

Unable to cast the type 'System.Int16' to type 'System.Object'.

I define the orderbyLambda as Expression<Func<E, object>> orderbyLambda

EDIT:

The real killer is if I do this:

 orderbyLambda:  p => new { p.ID }

It works... Why?