Here's a list of all known spider IP addresses in use:
I propose we take this data, put it into a robots.dat file with the LJ source, and provide a new option in ljconfig.pl:
$LJ::ACTIVE_ROBOT_BLOCK = 1;
Anybody want to work on this? Should be pretty easy, and the paranoid users out there will love you.