Hi! I'm a contributor to logjam, the GTK+ client. I've been on LJ for just over a year and have been interested in its technological sides since I joined. I've had jobs as a programmer (Perl, C, databases, unix) in the past but I find that as long as I can get away with it, programming is much more fun when I'm not payed to do it. Now, to business.
This is a proposal for getting entry comments sync into the protocol. I know this has been discussed before but has never been implemented yet, primarily because it's hard to get right. I'm sure my proposal isn't complete, but it tries to take into account the existing protocol style, load demands on the server, and client functionality. Please feel very much invited to discuss this here; I want to get it off the ground.
To give LJ clients a well-defined and efficient way to archive comments. Today clients can archive entries with syncitems and getevents(selecttype=syncitems); here I am concerned with extending this in a similar fasion to user comments.
The main consideration on the server side is to ensure the response doesn't get too big, but to make the limit useful to the client if it does. There are three mechanisms that I know of to tackle this problem; I adopted two of them. The mechanism this proposal does not use is chunking the response in equal-sized bits (like search results on Google, for instance, or viewing the last n entries in your friends list, or the several-paged view in huge discussions in news). What it does use is stateful sync (like getevents does) and folding threads and sending only information about the thread parent (like talkread.bml does). Assuming the hard limit on response count remains what it is today (500), this will probably always allow the server to give the client a full response, even if all the data it sends is folded. In the unlikely case where a really huge post gets lots of direct responses, there's the lightweight foldthread_n_parentid protocol response field on which I believe we can increase the limit substantially.
Issue: since currently the databases do not maintain comment sync information, this information will have to be fed into the tables sometime. There's no way about this, I'm afraid; I know Brad very recently commented he won't consider suggestions that affect very large tables, so the only workaround I can offer is to make populating these tables lazy, that is, have them generated the first time a client requests a comment sync. This will probably mean the very first request a user makes will fail with a timeout. This needs to be discussed; I'll leave it aside for now.
Issue #2: this feature will likely be limited to the user's own journal and communities he or she owns, for the same reasons getevents is.
The syncitems protocol mode already defines a comment type ("C") to be returned for comment items in the sync_n_item response field, so there's almost nothing to change in it. Except adding one request field:
synctype - An optional single value that specifies the server should only return items of the given type (either "L", "C", or "T"). If this is not set, the server sends items of all sorts.
The rationale behind this field is that if you are syncing only entries from a journal with 100 entries and 800 comments, the comments might clog your syncitems requests (there's a limit, currently of 500, of how many syncitems the server returns. In this scenario, since you're not syncing comments, you might have the server tell you there are 500 comments, and no more--and that's useless for you, since you'll never see the entries).
This is the proposed new mode for fetching comments, which mimics the existing getevents mode. The major difference between the two modes (apart from this one being for comments and not journal entries) is that getcomments will sometimes return folded "threadid" instead of the actual data, if otherwise the response would be too big. These threadids should be kept by the client and used on demand, e.g. when the user clicks on "expand thread".
Download comments to parts of the user's journal.
mode - The protocol request mode: getcomments
user, password, hpassword, truncate, prefersubject, noprops, lastupdate, beforedate, lineendings, year, month, day, lastn usejournal - akin to the same-name fields in the getevents mode.
setsync - Optional. If present and set to "1", marks downloaded information as archived by the client, even if it is selected with selecttypes other than "syncitems". The meaning of this field is only to prevent future syncs from downloading this data. This field need not be used when using the "syncitems" selection type; it is implicitly on then always.
selecttype - Here you choose how you want to select your comments. The possible values here fall into four groups:
- one - gets exactly one specific comment by itemid, without replies to it. If you use this selecttype and the request succeeds, you are guaranteed to get all the information that you asked for.
- thread, entry - using itemid as parent, return child entries of that itemid. (In the case of thread, return also the parent item itself since it is a comment; but not in the case of entry, where the entry itself was presumably already sent.) This selecttype returns a bunch of comments and/or threadids (see below), depending on the size of the output. That is, it does not guarantee to give you all the information you asked for, but it provides you with handles for getting more of it if you need.
- day, lastn - using the information in year/month/day (for day) or lastn (for lastn) to select a bunch of entries you're interested in, the server will respond as if you had made several getcomments requests of selectttype entry. Note that since this potentially selects a large number of comments, your chances of receiving threadids rather than actual comments increases, so in lastn you should keep howmany small.
- syncitems - get some number of items (which the server decides) that have changed since a given time (specified in the lastsync parameter). Not that because the server decides what items to send, you may or may not be getting everything that's changed. But unlike all of the modes above with the exception of one, this mode is guaranteed to give you comments and not folded threads.
success, errmsg - as in getevents.
events_count, events_n_itemid events_n_eventtime, events_n_event, events_n_subject - as in getevents
events_n_parentid - The id of the entry or comment to which this event is a reply to.
events_n_from - Optional. The userid of the author of this comment. Not sent if the comment is anonymous.
prop_count, prop_n_itemid, prop_n_name, prop_n_value - Used as in getevents (but used for: poster userpic, time, mood icon, ip address, etc.)
foldthread_count - Optional. If the server detected it was about to send too many comments in full, it "folds" some of the threads (at its own discretion) and sends only the leading comment in them, together with a note that this threadid is not "complete". If the user is interested in the thread, another request must be made to get it. (For very long threads this process has the potential of taking several iterations.)
foldthread_n_threadid - The itemid of a comment that leads a folded thread. Use this in a later getcomments/selecttype=thread request to unfold the thread.
foldthread_n_parentid - Optional. If the reply was so loaded that the server decided not to even send the the full event of the thread leader, it will supply the parentid of this thread so that the client at least knows where to place the fold.
That's it! Thanks for reading so far, and if you have any comments, please let me know about them! (So that I can one day download them? :-)