Convert 2 bytes into an integer

I receive a port number as 2 bytes (least significant byte first) and I want to convert it into an integer so that I can work with it. I've made this:

char buf[2]; //Where the received bytes are

char port[2];

port[0]=buf[1]; 

port[1]=buf[0];

int number=0;

number = (*((int *)port));

However, there's something wrong because I don't get the correct port number. Any ideas?