ByRef vs ByVal Clarification

I'm just starting on a class to handle client connections to a TCP server. Here is the code I've written thus far:

Imports System.Net.Sockets
Imports System.Net

Public Class Client
    Private _Socket As Socket

    Public Property Socket As Socket
        Get
            Return _Socket
        End Get
        Set(ByVal value As Socket)
            _Socket = value
        End Set
    End Property

    Public Enum State
        RequestHeader ''#Waiting for, or in the process of receiving, the request header
        ResponseHeader ''#Sending the response header
        Stream ''#Setup is complete, sending regular stream
    End Enum

    Public Sub New()

    End Sub

    Public Sub New(ByRef Socket As Socket)
        Me._Socket = Socket

    End Sub
End Class

So, on my overloaded constructor, I am accepting a reference to an instance of a System.Net.Sockets.Socket, yes?

Now, on my Socket property, when setting the value, it is required to be ByVal. It is my understanding that the instance in memory is copied, and this new instance is passed to value, and my code sets _Socket to reference this instance in memory. Yes?

If this is true, then I can't see why I would want to use properties for anything but native types. I'd imagine there can be quite a performance hit if copying class instances with lots of members. Also, for this code in particular, I'd imagine a copied socket instance wouldn't really work, but I haven't tested it yet.

Anyway, if you could either confirm my understanding, or explain the flaws in my foggy logic, I would greatly appreciate it.