Brad Fitzpatrick (bradfitz) wrote in lj_dev,
Brad Fitzpatrick
bradfitz
lj_dev

Active Robot Blocking

Since all of LJ is served dynamically, it's possible to do active robot blocking, rather than just using <meta> tags to politely tell them to go away.

Here's a list of all known spider IP addresses in use:

http://www.searchengineworld.com/spiders/spider_ips.htm

I propose we take this data, put it into a robots.dat file with the LJ source, and provide a new option in ljconfig.pl:

$LJ::ACTIVE_ROBOT_BLOCK = 1;

Anybody want to work on this? Should be pretty easy, and the paranoid users out there will love you.

Zilla bug:
http://zilla.livejournal.org/show_bug.cgi?id=394
Subscribe

  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    Help
  • 4 comments