Is it possible to disable authorization on one action in an MVC controller?

I have an authorization attribute on a controller, but I'd like to turn it off on one action. I created my own authorization filter and added "Anonymous" into the Roles list. In my filter I then return true if Anonymous appears in the role list.

However, it doesn't seem to get past the login page as if the controller authorization is pre-empting anything else.