|Active Robot Blocking
||[Dec. 16th, 2002|07:12 pm]
Since all of LJ is served dynamically, it's possible to do active robot blocking, rather than just using <meta> tags to politely tell them to go away.|
Here's a list of all known spider IP addresses in use:
I propose we take this data, put it into a robots.dat file with the LJ source, and provide a new option in ljconfig.pl:
$LJ::ACTIVE_ROBOT_BLOCK = 1;
Anybody want to work on this? Should be pretty easy, and the paranoid users out there will love you.