||[Jul. 12th, 2004|11:32 pm]
Have you (livejournal.com) ever considered setting up a database/web server just for bots? Most bots don't have the same performance requirements that a real user would need, and the timeliness of the data isn't very important either (1 day old friends list is fine for most bots, IMO).
There seems to be a growing collection of would-be projects that could benefit from it. You could even offer web hosting (to paid developers?) with direct (although read-only) access to various data. Maybe even have a donation-based feature where regular users could support their favorite memes.
It could have a latent replica of the production database all in one spot, so that queries didn't span multiple clusters. I suspect that there would be throttling to prevent overly popular projects from taking up too many cycles. MySQL and Apache support all of this with a minimum of configuration.
I realize that a lot of effort and consideration have gone into the various public interfaces, but server-side web apps could reside in your own data center. Thus, reducing much of the resource consumption that you're talking about. I might be oversimplifying this, but it seems that users really like memes and would appreciate the added features such an arrangement could provide.
(this was x-posted from my own journal)