C++ Templates: Partial Template Specifications and Friend Classes

is it possible to somehow make a partial template specification a friend class? I.e. consider you have the following template class

template <class T> class X{
    T t;
};

Now you have partial specializations, for example, for pointers

template <class T> class X<T*>{
    T* t;
};

What I want to accomplish is that every possible X<T*> is a friend class of X<S> for ANY S. I.e. X<A*> should be a friend of X<B>.

Of course, I thought about a usual template friend declaration in X:

template <class T> class X{
    template <class S> friend class X<S*>;
}

However, this does not compile, g++ tells me this:

test4.cpp:34:15: error: specialization of 'template<class T> class X' must appear at namespace scope

test4.cpp:34:21: error: partial specialization 'X<S*>' declared 'friend'

Is this not possible at all or is there some workaround?

The reason why I am asking is that I need a constructor in X<T*> that creates this class from an arbitrary X<S> (S must be a subtype of T).

The code looks like this:

template <class T> class X<T*>{
    T* t;

    template<class S>
    X(X<S> x) : t(&(x.t))  {} //Error, x.t is private
}

Now, the compiler complains, of course, that x.t is not visibile in the constructor since it is private. This is why I need a partial specialization friend class.