LiveJournal Development

Hey all, I think I'll take a little more initiative...

So, we need a good technical documentation system...

DocBook is gigantic and funny looking, using it might simply end up confusing people. Unless I can dig enough into it to find a suitable subset, and re-document that. Which might be twice as much work as it's worth.

I'll write a DTD for XML-based lj docs, and I believe with a few examples, and with the common knowledge of HTML, this should be simple enough at first.

I'll write out some abstract scripts, one to validate and produce (as much as possible) coherent and helpful errors for the docs as they're submitted... and another to convert the docs to HTML. I'll write one or two other converters, but then I'll sit here and whine until someone else does the necessary code to convert to the other file formats other people want.

So, king of all suggestions, what is your wishlist for sections/tags in an lj documentation system?

I was rolling around in bed, unable to sleep (what's new?), when I thought..... "why the hell haven't I just released all the livejournal source?"

I remember having a few reasons in the past, but I honestly can't remember them now, or they're no longer applicable. There is one potential negative effect (which I'll discuss in a second), but everything other case of how the source would be handled would be beneficial.

Here's the one negative thing.... security. I don't believe in "security through obscurity", but I do believe that security holes and serious bugs tend to be found less in code that's a big black box. Especially for somebody trying to find a hole. Before, if somebody wanted to hack livejournal it'd mean trying to root the box, then causing damage. However, now it'd be much easier... download the code and find that one place where I did something stupid and never caught before.

So---- here's my plan: I want to have a bunch of you do code reviews of all the source before I release it. Let's say I release it in a week.... Monday, March 5th. Does that give all of you Perl hackers enough time?

I want tons of comments, itemized by file and line number... anything that looks suspicious. Please focus on security problems and bugs only, not feature enhancements or changes. Keep a separate file for those, and we can discuss them in a week. For now, all I care about is making sure it's tight enough to release to the world.

Then, I want to work on an INSTALL file, since installing livejournal is no easy task.

Then --- I want at least one of you, ideally several, to actually run all of LiveJournal on your own machines for testing/development.

Also, give me suggestions for licenses, since I hate them but realize I should stick one on here. As far as I know, the GPL is useless for web applications, since you can take the code, modify it, run it on your site, and never have to release your changes because you never released a binary. So, X/BSD/MIT license? Or, should I write my own, in English? Don't ask me what I want out of the license, because I really don't even care.... I want to hear what you think I should care about, and what license I should then use.

So yeah, if you're somewhat competent with Perl and want to review the source, let me know. I know they say Perl is a write-only language, but my Perl is a lot easier to read than most, but perhaps I'm biased. :-)

Also, if anybody has any reasons why perhaps I shouldn't just release all the code, now's the time to state your case, and let everybody here debate it.

The reason I'm doing this, if anybody's curious, is because I'm getting increasingly stressed out with so much to do, and I realize that the only way I'll get any help is if people can be running development livejournal servers on their own machines.

After the source is open, I'll setup rsync/CVS/cvsweb, etc...

I'm still going to be the sole authority on what goes into the main tree, though. I'm incredibly picky about bad code and bad design, and I'll be honest with you if I don't like something, but I think everything will be fine. I hope people will take up the task of documenting, too... some of the LJ API functions are documented, but others aren't. We need shitloads of documentation, really.

This'll be fun...

I've been reading a lot of people thinking that when the LiveJournal source is released they'll be able to just drop it in on their own server and have a easy-to-customize version of LiveJournal running on their own machine.

No, it won't be that easy.

That's what the MyLJ project is about.... all the tools, clients, servers necessary to do that yourself, off your own server, or a combination of your server and another, like livejournal.com's.

Releasing the source, however, may expedite development of MyLJ ... we'll see.

So far the code contribution from lj_dev people has been minimal, understandably. However, from the enthusiasm and helpfulness I see here, I suspect things will speed up shortly.

I just hope that opening the source doesn't stall MyLJ development, since people might think MyLJ isn't necessary if all the code is open. But the current code is very livejournal specific. It'd need a lot of changing around to get it to be more general. However, as MyLJ development is currently pretty slow, it can't get much worse.

Also, you won't need huge servers to run the code.... you won't be doing 50+ requests/second like LiveJournal. You'll be doing one request every 5 seconds. Your spare P90s or whatnot will handle it fine.

The latest and greatest version of my LJ Sync program is up and available. The details are in this entry. Bang away with it and let me know what you think. Still working on proxy support, sorry it's not in yet.

Hey all I'm trying to determine whether or not there are still issues with my verion of LJSync or if it's a server response problem. If you can download it and just tell me whether or not it syncs up for you that would be great. If it doesn't work you will know it, you will get an invalid property value and it will bomb. I'm 4/5 on journals that I've sync'd but I've heard reports that it isn't working with their specific journal. So unless some people are doing some funky things with their journals I don't know what to do. Can you test it out and reply to this entry with your results. A simple "yes it worked" or "no it didn't" would be fine.

we really need to get the new memory for stan soon (they shipped it today I think, finally). once stan's up and can handle web requests, i want to get kenny being a read-only slave db server. it has nice scsi disks, and is the fastest of all the machines. it'll still do img.livejournal.com and ftp.livejournal.com also, and be the primary place where i login, tweak code and administer things.

cartman's the main bottleneck lately... database is locked way too often. the load's about 8.... CPU pegged. too many reads and writes.

also, I need info from you guys about the large file support.... that's going to be an issue pretty soon here:

[root@lj-cartman livejournal]# ls -l | sort -k 5 -n | tail -10
-rw-r--r-- 1 root root 49452684 Feb 26 17:54 logprop.MYD
-rw-r--r-- 1 root root 66414592 Feb 26 17:54 log.MYI
-rw-rw---- 1 root root 67756032 Feb 26 17:54 syncupdates.MYI
-rw-r--r-- 1 root root 67767457 Feb 26 17:54 talk.MYD
-rw-r--r-- 1 root root 76336137 Feb 26 17:54 log.MYD
-rw-r--r-- 1 root root 112238592 Feb 26 17:54 talk.MYI
-rw-r--r-- 1 root root 134941996 Oct 17 01:03 talkpost.MYD
-rw-r--r-- 1 root root 275162808 Feb 26 17:53 userpicblob.MYD
-rw-r--r-- 1 root root 434013316 Feb 26 17:54 talktext.MYD
-rw-r--r-- 1 root root 1,509,416,996 Feb 26 17:54 logtext.MYD

Cartman is linux 2.2.18, glibc-2.1.3, ext2fs. Can that do LFS?
Kenny is freebsd 4.2-stable.... Can that do LFS? I think it can.