Brad Fitzpatrick (bradfitz) wrote in lj_dev,
Brad Fitzpatrick
bradfitz
lj_dev

Replacing syncitems

I want to replace the syncitems tables/protocol mode with something else.

Syncitems takes a lot of disk space and nobody uses it because no clients exist because it's too complicated.

I propose two new logprops:

postrevstring
comrevstring

Both are opaque revision identifiers that backup clients can use to decide if they already have the data cached locally. On an editevent, we just update the postrevstring. It could be anything... incrementing number, unixtime, or md5 of all data. I'm leaning towards md5 there.

Comrevstring would be updated whenever an event took place that edited comments on a post... new comment, unscreened, deleted, etc. The format of this would probably be "unixtime-remoteid" to avoid needing to read beforehand and do locking.

But regardless of their formats, all backup clients would need to do is:

-- fetch all post IDs & rev strings (one request)
-- for each post with a new postrevstring, fetch it. (could make new selectmode for getevents to let people retrieve multiple at once)
-- for each post with a new comment string, retrieve all comment meta-data on post. (new API mode).
-- retrieve comment text for all not already cached (comments can't be edited, so no revstring needed there)

I have an ActiveState Perl Developer Kit license now, so I can compile the Perl backup client for Windows.

I'm also about 95% done porting S2 from Java to Perl, so the backup client can later include the S2 compiler and runtime to generate client-side HTML from the backup archives.

But syncitems should go. Nobody's using it anyway, right?

Discuss.
Subscribe
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 7 comments