December 16th, 2002

Active Robot Blocking

Since all of LJ is served dynamically, it's possible to do active robot blocking, rather than just using <meta> tags to politely tell them to go away.

Here's a list of all known spider IP addresses in use:

http://www.searchengineworld.com/spiders/spider_ips.htm

I propose we take this data, put it into a robots.dat file with the LJ source, and provide a new option in ljconfig.pl:

$LJ::ACTIVE_ROBOT_BLOCK = 1;

Anybody want to work on this? Should be pretty easy, and the paranoid users out there will love you.

Zilla bug:
http://zilla.livejournal.org/show_bug.cgi?id=394