User Comments
It appeared to be a home-grown bot Gabriel, using LWP. 20+ IP addresses were used during the spidering. I should have used whois on the IPs, but I didn’t. I was panicking as I tried to get the server back up. |
My current throttle profile is as follows:
Whilst everything seems to be working okay presently, I’m not sure if the above policy is suitable. Any advice will be gratefully received. |
Who was it? Lately, MSNbot has requested pretty much all of my content in a matter of hours. At least I know that its searches will return correct paths, unlike Google who still returns URI which haven’t been valid for a year or more!