Morgan (boobay) wrote in lj_dev,
Morgan
boobay
lj_dev

stuuupid robots (Teoma)

I just launched my own Travel Community based in parts on LJ and in the last couple days Teoma crawled thousands of pages from the Months view for months/years which do not exist, e.g. "GET /users/moonwha/1846/05/

today it went down to the 18th century even (17xx), and for some users even up into the 22th century, e.g /users/rigsby3/2161/10/ - the more users signed in the worse the problem got (right now teoma still crawls 1-2 of these urls per sec) google+grub don't seam to do that
The problem is that htdocs/view/index.bml always shows arrows for the prev + next months - even if there are no entries anymore for a user, I created a quick "fix" to:
a.) only show arrows if there are more (prev|next) entries and
b.) to skip months (e.g. jump from Jan directly to May if there are no entries in between.

in htdocs/view/index.bml:89

my $prev = "";
my $next = "";
my $db = LJ::get_cluster_reader($u);
$sth = $db->prepare("SELECT year, month from log2 where journalid=$u->{'userid'} and ((year<$year) or ((year=$year) and (month<$month))) order by eventtime desc limit 1");
$sth->execute;
if (($prev[0], $prev[1]) = ($sth->fetchrow_array)) {
$prev = "";
}
$sth = $db->prepare("SELECT year, month from log2 where journalid=$u->{'userid'} and ((year>$year) or ((year=$year) and (month>$month))) order by eventtime limit 1");
$sth->execute;
if (($next[0], $next[1]) = ($sth->fetchrow_array)) {
$next = "";
}
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 5 comments