Problem with Factory_girl, association and after_initialize

I have a Family class so defined:

class Family < ActiveRecord::Base
  after_initialize :initialize_family
  belongs_to :user
  validates :user, 
       :presence => true

  validates :name,   
       :presence => true,          
       :length => { :maximum => 30 },
       :format => { :with => /\A[a-zA-Z0-9\-_\s\']+\z/i}

  def initialize_family
    if self.name.blank? && self.user
        self.name = "#{self.user.profile_full_name}'s Family"
    end
  end
end

In my factories.rb I have:

Factory.define :family do |f|
   f.association :user, :factory => :user
end

In my family_spec.rb I have

let(:family) { Factory(:family) }

But this fails with:

1) Family is valid with valid attributes
     Failure/Error: let(:family) { Factory(:family) }
     ActiveRecord::RecordInvalid:
       Validation failed: Name can't be blank, Name is invalid, Languages can't be blank, Languages is too short (minimum is 1 characters)
     # ./spec/models/family_spec.rb:8:in `block (2 levels) in <top (required)>'
     # ./spec/models/family_spec.rb:10:in `block (2 levels) in <top (required)>'

Using the debugger I can see that when after_initialize is called self.user is nil. Why is this happening? If I call the family with create or new everything works fine.

Thanks for any help.