January 23rd, 2003

Idea 1/2: syndication cluster

New servers available for pickup on Monday (burn-in test over the weekend). In the meantime, I'm scratching/pounding my head over to make the site faster.

I have two ideas. Here's number 1:

-- syndication articles are always posting and always deleting. this causes lots of writes and lots of fragmentation, causing new posts from real users to fill into holes on many-day-old disk pages, killing cache hit rates. MySQL (unlike "real" databases) doesn't let you group data records together physically on disk based on certain indexes.

-- syndication content isn't necessary to have super reliable.

-- we could have a cluster just for syndication without any slaves, without tons of redundant hardware. the advantage will be the real database clusters won't get so fragmented. the deleted articles (over 2 weeks old) will just be replaced with the new articles.

This is so easy to do, it's worth trying.

But first I need to make moveucluster.pl allow users to be migrated between clusters. Currently the tool only supports moving from "cluster 0".

Idea 2/2: more blatant "slow down" on friends page reloading

There is currently a capability setting for often a user can reload their friends page. But we're not too loud about violating it.... we do 304 Not Modified if possible, otherwise we just give them their friends page up to that last valid point in time when they were allowed to reload.

So if there is a new entry every minute and they're reloading every 30 seconds, they'll only see new stuff every 6 reloads, not every 2, if their allow reload is 3 minutes.

Unfortunately, this doesn't save CPU, bandwidth, or DB I/O. I think we should instead say (to free users), on a white blank page "Whoa, reloading kinda fast there, aren't we? If you're this addicted, go get a paid account, otherwise reload in 7 minutes and 24 seconds" with a "503 Out of Resources" or something, so it's not cached/indexed on accident. The detection would be done on how quick logged-in users are reloading _any_ friends page, or if nobody's logged in, then the IP address, but IP addresses would have slightly higher limits, to minimize false positives from people coming through proxy servers. Or hell, maybe we go back to giving everybody (even not-logged-in users) a random cookie (ala mod_unique or whatever).

But the point is: people who are reloading every 2 minutes should be told they're being annoying, rather than appeasing them at the cost of site performance.

I'm sure many people disagree with me here, but I get angry and desperate when the site's slow. Could this at least be acceptable in bad times, like now? We could lower the threshold time when things are good.... bitching only when people reload, say, every 30 seconds or something.