MVC Attributes on Controllers and Actions

Is there a way to add an Attribute on the Controller level but not on a specific action. For example say if i had 10 Actions in my Controller and just 1 of those Actions does not require a specific attribute I created.

[MyAttribute]
public class MyController : Controller
{
    public ActionResult Action1() {}
    public ActionResult Action2() {}

    [Remove_MyAttribute]
    public ActionResult Action3() {}
}

I could potentially move this Action into another controller (but dont like that) or I could apply the MyAttribute to all actions except from Action3 but just thought if there is an easier way?