Brad Fitzpatrick (bradfitz) wrote in lj_dev,
Brad Fitzpatrick

Projects status

Quick update on the status of a bunch of big projects, since it might seem nothing exciting's happening, even though a ton of exciting stuff is happening:

Load Balancing -- we've written our own load balancing system. I originally suggested that F5 add it to their BIG-IPs (which we use) and they said they loved the idea, but wouldn't be able to add it for quite some time. Basically, what happens is that we're making our internal webservers advertise their number of free children whenever it changes. The proxy machines monitor that, and using mod_rewrite's external rewrite engine support and mod_proxy's reverse proxy, does a weighted random selection to go to a machine that's known to have free children, or waits for one. End result: perfect load balancing, and no more waiting many seconds in a listen queue just to talk to a webserver. We have one of our front-end proxies doing this new method already and it's amazing... load is spread much more evenly than it was before. (before friends pages and easy requests were being intermixed and the load balancer couldn't make good decisions....) Almost all work done by avva. It's all open source but not documented yet. It will be.

Blob Server -- Before evan left, he finished the media/"blob" server support, so we can efficiently store lots of big files for users. The userpicblob table will go away, and everything will use the blob server. (there are two modes: local and remote... people with small installations will use the local mode, right on the file system).

Phone Post -- Evan also checked in post-by-phone support a couple weeks ago. It uses Asterisk. It's in CVS and works great... if you know anything about Asterisk, check it out! We'll put this live once our new storage hardware arrives (in less than 2 weeks, we're told). This depends on the blob server.

Memcached -- nothing exciting. Pretty stable. Been working with the libevent and epoll maintainers and they've been fixing bugs, so it's even more stable. We'll be testing slab class reassignment soon, so it's even more hands-off than it already is. Currently you should restart memcache nodes when the distribution of object sizes changes... but we're making that automatic so it's self-tuning.

Hiring -- We're hiring more programmers, so development will be speeding up even more.


I'm thinking it'd be nice to get a community member to gather technical questions once or twice a week and do phone calls with one of us (probably me) and then post a review in lj_dev. Sadly, my wrists/arms kinda hurt lately, so I've been trying to take it easy on the typing.

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded