Re: Some thoughts & meeting time

[Roger Dingledine <arma@mit.edu>]

In message <Pine.BSF.4.05.9811102155440.8935-200000@shell5.ba.best.com>, 
 aturner@best.com writes:
>Hey people:
>
>First, mark your calanders Sunday 11/15, 9am PST (12 noon Eastern) for our
>next LKB IRC chat on #seul (cran.seul.org).  Come one, come all!  If you
>don't like it/can't make it, let me know tomorrow (11/11).  Again, I'd
>like at least 2 people from each group, everyone would be better.

Actually, I vote Saturday at 9am PST (12pm EST (6pm German time)), since
I'm going to be busy on Sunday and busy past 3:30EST on Saturday. If
people don't like that, then I vote one hour later on Sunday, so Dana
can make the meeting on time (even tho I won't be there). Lenz mentioned
that he ought to be able to make either Saturday or Sunday.

>Be ready to discuss:
>1) Will merge into one group?  (Jason has put the offer to us, we haven't
>accepted yet)

I vote yes. I think it would be dumb not to.

>2) What you as a person want to do.  Be able to commit to a certain # of
hours/week.
>
>3) Short term goals and needs (let's leave the long term distribution
>method off the discussion list until after we have a site live)
>
>4) Any new business
>
>Roger and I had a good discussion this evening regarding a number of
>issues.  I'm writing a summary here, but those who are looking for a
>sleeping aid, can read the IRC log which I've attached.  
>
>One of the things we discussed was the concept of a front-end vs.
>back-end.  Ie, will a search engine or index be the primary user
>interface?  We seem to agree that the search engine (ala ht://dig or the
>like) will be the primary interface the users use.
>
>We discussed the serious problem of having multiple/redundant servers and
>how to fail over.  Both Roger and I agree that the DNS caching issues are
>major and special consideration will need to be made.  Three potential
>solutions are:

Err..here's my description of the situation.

This is our long-term goal:
The LinuxKB network will consist of:
* several KSs (knowledge servers), which are the machines that contain the
  actual documents
* several SSs (search servers), which are the machines which interface to
  the users, perform search queries, etc. Each SS will also have browsable
  indices for the KB (though we might put the browsable part on the KS).

Each KS stores its entries in MySQL, and periodically generates a set of
static html pages from those entries. These static pages are the
documents that it serves to the public. Also periodically, these static
html pages get propagated to each SS so they will be able to do searches
based on the most recent material. I'm sure there are incremental update
algorithms so only the changes in the static html pages need to be propagated
to the SS. When a user performs a query at an SS, it will respond with a
set of links to the appropriate documents on the appropriate KS.

All of that is quite simple. Now some slightly trickier extensions: even
though data may be distributed among a set of KSs, every KS will be able to
answer any document request. If the document is not actually stored at that
KS, then it will fetch it from the appropriate KS and then serve the
document to the user. This can be made more efficient through caching at
each KS.

The difficulty that Aaron is describing is that we don't know which
machine should be returned when you ask for "www.linuxkb.org" -- which
SS should serve the main queries, and what happens when that one goes
down? (The whole point of multiple machines is massive redundancy.)
There are a couple solutions that I think might work (or at least
suffice), but I still need to hash them out a bit farther.

The good news? This is LONG term. You don't have to worry about it now.
The better news? We can move in this direction while still doing the
right thing short-term.

The short-term plan:
We have one KS and one SS, and they're on the same machine.
Finish the mysql tables for how to store the data. Write the scripts for
generating the static html pages from the mysql tables. Then somebody needs
to set up some sort of search engine that will be able to index all the
static html pages (Lenz said he could probably set that up. Else Aaron
can, I think). Also, Aaron will need to set up his cgi user-interface
scripts to interface with the submission + query system, so we can grow
the KB. Once we have
* The tables
* the glue to turn tables into static html pages
* the thing that searches static html pages
* a front-end for people to run searches and submit documents

then we have enough to start.
Once we've got something up and working, then some of us can start
maintaining it and patching it so it'll last longer, and the rest of
us can start looking toward the horizon. (And we can start gaining
workers.) But as Aaron said, let's leave discussion of the long-term
plans off the list until we have a live site. This was a summary to
give people an idea of what they should be keeping in mind when they're
designing.
Am I missing anything for the short-term plan? (Can we expect a draft
of the mysql tables soon, Jason?)

Thanks,
--Roger