Adding SSH Key to authorized_keys: permission denied(publickey)

• I have an id_rsa and id_rsa.pub on my computer also tied into my BitBucket account.
• As I understand I can use this public key anywhere I please (as long as my private matches).

The Problem: I cannot figure out how to get any server to accept the public key, do you see what I've done wrong?

 $ cat id_rsa.pub >> authorized_keys

 $ service ssh restart  (I suppose this isn't needed)

 $ git pull origin master

 $ Permission denied(publickey)

What am I doing wrong? I've been stuck for days.