December 16th, 2002

Active Robot Blocking

Since all of LJ is served dynamically, it's possible to do active robot blocking, rather than just using <meta> tags to politely tell them to go away.

Here's a list of all known spider IP addresses in use:

I propose we take this data, put it into a robots.dat file with the LJ source, and provide a new option in


Anybody want to work on this? Should be pretty easy, and the paranoid users out there will love you.

Zilla bug: