Custom route handlers in ASP.Net WebAPI

I'm able to successfully register a custom route handler (deriving from IRouteHandler) inside global.asax.cs for a Web API route ala:

        routes.MapHttpRoute(
            name: "DefaultApi",
            routeTemplate: "{client}/api/1.0/{controller}/{action}/{id}",
            defaults: new{id = UrlParameter.Optional}
        ).RouteHandler = new SingleActionAPIRouteHandler();

However I can't seem to find a way to do this when trying to setup an in memory host for the API for integration testing, when I call HttpConfiguration.Routes.MapRoute I'm not able to set a handler on the returned IHttpRoute.

Should I be doing this differently (for instance by using a custom HttpControllerSelector)? I'd obviously like to do it the same way in both cases.

Thanks, Matt

EDIT:

So what I ended up doing was basically following the advice from below, but simply overriding the HttpControllerDispatcher class as follows:

public class CustomHttpControllerDispatcher : HttpControllerDispatcher
{
    public CustomHttpControllerDispatcher(HttpConfiguration configuration) : base(configuration)
    {
    }
    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        // My stuff here

        return base.SendAsync(request, cancellationToken);
    }
}