Brad Fitzpatrick (bradfitz) wrote in lj_dev,
Brad Fitzpatrick

Idea 1/2: syndication cluster

New servers available for pickup on Monday (burn-in test over the weekend). In the meantime, I'm scratching/pounding my head over to make the site faster.

I have two ideas. Here's number 1:

-- syndication articles are always posting and always deleting. this causes lots of writes and lots of fragmentation, causing new posts from real users to fill into holes on many-day-old disk pages, killing cache hit rates. MySQL (unlike "real" databases) doesn't let you group data records together physically on disk based on certain indexes.

-- syndication content isn't necessary to have super reliable.

-- we could have a cluster just for syndication without any slaves, without tons of redundant hardware. the advantage will be the real database clusters won't get so fragmented. the deleted articles (over 2 weeks old) will just be replaced with the new articles.

This is so easy to do, it's worth trying.

But first I need to make allow users to be migrated between clusters. Currently the tool only supports moving from "cluster 0".

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded