Brad Fitzpatrick (bradfitz) wrote in lj_dev,
Brad Fitzpatrick
bradfitz
lj_dev

recent_*

This is probably not of much interest to people, but I want a place to be able to point people in the future should I need to.

We now have two new tables: recent_logtext and recent_talktext. They are the same format as logtext and talktext but they use 4 byte pointers instead of 5 byte pointers. In other words, they're intended to always be under 4 GB in size.

Any inserts/updates happen to both tables.

recent_* are replicated to our dummy database slaves, but not logtext and talktext.

recent_* tables will be cleaned every night or so, removing everything older than a week.

Applications will no longer query logtext/talktext directly to get data... instead they'll go through an API which first tries recent_*, and if it doesn't get everything, then goes on and gets the remaining ones from the real tables (on the master database).

The whole point of this is to load-balance the database. Those two text tables are the biggest tables and cause by far the most disk activity and overall load on the database (especially since they contain variable-length records).

This should make cartman's job a ton easier.

Current Status:
config options: now in CVS, ljconfig.pl
recent_logtext writing: now in CVS, ljprotocol.pl
recent_logtext reading: not done
recent_talktext writing: not done
recent_talktext reading: not done
ljmaint script to clean recent_: not done

ljprotocol.pl is patched and in CVS to do the recent_logtext stuff.
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 7 comments