Regex - simple phone number

I know there are a ton of regex examples on how to match certain phone number types. For my example I just want to allow numbers and a few special characters. I am again having trouble achieving this.

Phone numbers that should be allowed could take these forms

5555555555
555-555-5555
(555)5555555
(555)-555-5555
(555)555-5555 and so on

I just want something that will allow [0-9] and also special characters '(' , ')', and '-'

so far my expression looks like this

/^[0-9]*^[()-]*$/

I know this is wrong but logically I believe this means allow numbers 0-9 or and allow characters (, ), and -.