For some reason, when certain bots visit the site, generating a url with the UrlHelper.Action method raises a null exception from System.Web.HttpServerVarsCollection.Get. I've done some debugging and the call stack originiates with an attempt to get the 'HTTP_X_ORIGINAL_URL' from the HttpContextBase.Request.ServerVariables collection.
If I visit the same address directly from a browser - no problem. The page is server and no error is logged. It only seems to occur when visited by a bot.
Not sure if it's relevant or not, but the site was just migrated to IIS 7.5. Still using .NET 2.0 in Integrated mode.
Looking at the code as reversed by Reflector the only place where a null exception can occur directly in the Get method is the call to this._request.FetchServerVariables. As if the complete request was not set up properly.
Has anyone else faced this problem or discovered a workaround? Why would the request be set up differently when visited by a bot?
UPDATE: Some additional debugging has shown that the HttpServerVarsCollection was Disposed, along with it's parent HttpRequest object. The question now is - how can the Request object returned by HttpContext.Current be exposed before the request is complete?
HttpServerVarsCollection.Get Method
public override string Get(string name)
{
if (!this._populated)
{
string simpleServerVar = this.GetSimpleServerVar(name);
if (simpleServerVar != null)
{
return simpleServerVar;
}
this.Populate();
}
if (this._iis7workerRequest == null)
{
return this.GetServerVar(base.BaseGet(name));
}
string serverVar = this.GetServerVar(base.BaseGet(name));
if (string.IsNullOrEmpty(serverVar))
{
// Only place null reference can happen
serverVar = this._request.FetchServerVaria