Generic type pointers? and casting byte array to given type of generic?

Okay the basic idea what I'm trying to do is, converting byte array to something like short or int etc. etc.

A simple example might be:

        unsafe
        {
            fixed (byte* byteArray = new byte[5] { 255, 255, 255, 126, 34 })
            {
                short shortSingle = *(short*)byteArray;
                MessageBox.Show((shortSingle).ToString()); // works fine output is -1
            }
        }

Okay, so what I'm really trying to do is, make an extension to Stream class; extended read and write methods. I need help at the following code:

unsafe public static T Read<T>(this Stream stream)
        {
            int bytesToRead = sizeof(T); // ERROR: Cannot take the address of, get the size of, or declare a pointer to a managed type ('T')
            byte[] buffer = new byte[bytesToRead];
            if (bytesToRead != stream.Read(buffer, 0, bytesToRead))
            {
                throw new Exception();
            }
            fixed (byte* byteArray = buffer)
            {
                T typeSingle = *(T*)byteArray; // ERROR: Cannot take the address of, get the size of, or declare a pointer to a managed type ('T')
                return typeSingle;
            }
        }

        unsafe public static T[] Read<T>(this Stream stream, int count)
        {
             // haven't figured out it yet. This is where I read and return T arrays
        }

I feel like I have to use pointers for speed because I will be working on writting and reading data from streams like NetworkStream classes. Thanks for your help!

EDIT:

And while I try to figure out how may I return T arrays, I've faced with this problem:

unsafe
        {
            fixed (byte* byteArray = new byte[5] { 0, 0, 255, 255, 34 })
            {
                short* shortArray = (short*)byteArray;
                MessageBox.Show((shortArray[0]).ToString()); // work