jQuery selecting and filtering elements inside a div

I have a problem with selecting and filtering elements inside a div.

HTML :

<div id="wrapper">
    <input type="text" value="you can edit me">
    <input type="button" value="click me">
</div>

jQuery :

$("#wrapper").children().click(function() {
    alert("hi there");
});

The problem is I get alerted every time I click anything inside the div.
But my requirement is to alert only when the user clicks on the button.
I know that filtering the elements in jQuery is using :button

This is what I have tried :

$("#wrapper").children(":button").click(function() {
    alert("hi there");
});

and

$("#wrapper").children().filter(":button").click(function() {
    alert("hi there");
});

It didn't work

Anyone know how to do this?