EF 4.1 SaveChanges not updating navigation or reference properties

I'm using code first and can add records without a problem. The database is created properly and is seeded without any problems. But calling SaveChanges() in my Edit action only updates the entity, not any of the navigation or reference properties.

A simplified version of my model:

public class Contact : IMyBaseObject
{
    public Contact()
    {
       this.Delete = false;
       this.ContactTypes = new HashSet<ContactType>();
    }

    public int Id {get; set;}

    public string Name {get;set;}

    public bool Delete {get;set;}

    public virtual ICollection<ContactType> ContactTypes { get; set; }

    public virtual USState USState { get; set; }

}

public class ContactType : MyBaseObject
{
    public ContactType()
    {
    }

    public int Id {get; set;}

    public string Name {get;set;}

    public virtual ICollection<Contact> Contacts {get;set;}
}

public abstract class Territory : MyBaseObject
{
    public int Id {get; set;}

    public string Name {get;set;}

    public string Code {get;set;}
}

public class USState : Territory
{
    public USState()
    {
    }

    // Navigation properties
    public virtual ICollection<Contact> Contacts { get; set; }
}

I won't include the code, but I do have some custom model binding going on. My Edit action (using MVC 3) was not being populated with the ContactType or USState properties. The binding is properly returning a fully populated Contact object with the correct values from the form.

If I understand EF 4.1 properly, I should only have to do the following to persist changes to the database when saving my Contact object:

if(ModelState.IsValid)
{
    context.Entry(contact).State = EntryState.Modified;
    context.SaveChanges();
}

However, only primitive properties are being updated when I do that. I have left out a few things in the above example: I'm using a transaction, a try catch block, and check