Some thoughts & meeting time

["Aaron D. Turner" <aturner@best.com>]

Hey people:

First, mark your calanders Sunday 11/15, 9am PST (12 noon Eastern) for our
next LKB IRC chat on #seul (cran.seul.org).  Come one, come all!  If you
don't like it/can't make it, let me know tomorrow (11/11).  Again, I'd
like at least 2 people from each group, everyone would be better.

Be ready to discuss:
1) Will merge into one group?  (Jason has put the offer to us, we haven't
accepted yet)

2) What you as a person want to do.  Be able to commit to a certain # of
hours/week.

3) Short term goals and needs (let's leave the long term distribution
method off the discussion list until after we have a site live)

4) Any new business

Roger and I had a good discussion this evening regarding a number of
issues.  I'm writing a summary here, but those who are looking for a
sleeping aid, can read the IRC log which I've attached.  

One of the things we discussed was the concept of a front-end vs.
back-end.  Ie, will a search engine or index be the primary user
interface?  We seem to agree that the search engine (ala ht://dig or the
like) will be the primary interface the users use.

We discussed the serious problem of having multiple/redundant servers and
how to fail over.  Both Roger and I agree that the DNS caching issues are
major and special consideration will need to be made.  Three potential
solutions are:

1) Get access to hardware which does this (Cisco Local Director, Resonate,
etc)

2) Place all the machines locally on the same LAN and set them up to
masqurade as needed.  (there is a howto on this)

3) ???


We determined that we have to maintain 3 data-sets:
SQL DB generated via web forms
Static HTML pages generated from SQL DB
Search engine DB generated from Static HTML pages

And that this should be modular.  Initially all would likely be on the
same machine, but moving the search engine to it's own machine will be
needed.

I also brought up the possibility of using socket calls to distribute the
user requests across multiple/redundant machines.


We discussed the long-term nntp issues.  One thing we agreed on is the
need to find an expert in INN to create a RPM to allow 'mirror' site
easily setup a mirror.  More thought on the underlying protocol needs to
be made however- a simpler system may be possible (perhaps a pull system
instead of a push).  INN will work for our purposes, but a simpler
solution may be prefered.

That's it for now.

-- 
Aaron Turner           | Either which way, one half dozen or another. 
aturner@pobox.com      | Check out the Red Hat Linux User's FAQ Online!
www.pobox.com/~aturner | http://www.pobox.com/~aturner/RedHat-FAQ/
All emails from this account are PGP signed.  Lack of a signature is "bad".
PGP Key fingerprint = FB E1 CE ED 57 E4 AB 80  59 6E 60 BF 45 1B 20 E8

<arma> good evening.
<rhlufaq> Hey Roger.
<arma> you should pick a time for the weekend meeting, and float it to the list.
<rhlufaq> Yeah, I was thinking that too... been kinda busy lately and I've got another weekend class thing to plan.
<rhlufaq> I'm going to send something out tonight though.
<arma> ok
<rhlufaq> Is Dana really here, or is he just left-on?
<arma> i'm dealing mostly with seul-pub and seul-edu these days.
<arma> dana is left-on, but he is here elsewhere if you want me to get him for you.
<arma> he's on a mud that he hosts on his computer
<rhlufaq> When he's got a few minutes to burn, I want to get more info on that project he was talking about.
<arma> which one? the linux news project?
<rhlufaq> Not important enough to pull him off a mud- we all need a break.
<rhlufaq> Yeah, LNP.
<arma> well, he's not mudding..he's the "guy in charge" there. he's just there talking.
<arma> but in any case, the LNP is my project. i can answer questions.
<rhlufaq> Well I was wondering what kind of controls you/he invisioned for the "mirrors"... I've had some really good mirrors, and some really bad ones that I've had to 'kill' because they hurt the project.
<arma> ok..the concept of mirror is not what you think.
<arma> there is no primary site. they are all mirrors of each other.
<arma> changes propagate when they're made. it's push, not pull.
<arma> (or at least, it should be thought of that way. it can be implemented either way, provided you query frequently.)
<arma> if you want more background, you can read the lu-news archives (http://www.seul.org/archives/lu/news/)
<arma> there's rather a lot of reading, tho
<rhlufaq> I understand that- sorry for the poor choice of words.  What I'm trying to get at, is keeping the quality there.  
<rhlufaq> I guess I had planned some standardization= something this method doesn't provide
<rhlufaq> It's a good thing that allows admins to have control, but also a bad thing
<arma> urm...i don't understand
<arma> are you saying what i just described doesn't provide standardization?
<arma> or that it might increase load too much?
<rhlufaq> Standardization- each site admin can do what they want with the info and present it in their own way.  No control is placed regarding who has what.  Using the DNS
<rhlufaq> analogy, you have primary, and multiple secondaries... but unlike with DNS, people/compainies won't have the same...
<rhlufaq> interest in creating secondary/backup sites.
<arma> hrm
<rhlufaq> I guess that's kinda really two issues.  #2 is probably easily solved, but #1 is still an issue.
<arma> that's not true. the interest is that people share their news with you,
<arma> and you share your news with people.
<arma> can you clarify your concern?
<rhlufaq> But the KB is different.  News (only interesting to some people) vs. Knowledge (interesting to all users)
<arma> no, that's not how it works
<arma> news has categories. various people are interested in various categories.
<arma> same with knowledge.
<arma> unless you mean that not everybody wants to read news. in which case, yeah...the kb will have more users.
<rhlufaq> But I'm willing to go to a site devoted to one kind of news; knowledge is so intertwined that I need to have access to all the info from each/one site.
<arma> true. there is an issue for how to make sure all sites know about what lives in the db
<arma> there could feasibly be a server that yields knowledge about filesystems,
<arma> and a server that yields knowledge about linux advocacy,
<arma> and a server that has an index of all the known servers (or a subset) and serves search requests (which are then handed to the relevant dedicated server)
<rhlufaq> And allowing them to quickly access the info from remote machines... Having one server devoted to each area creates a boat-load of problems IMHO.
<rhlufaq> Now you need some governing body to manage who gets what.
<arma> sure
<arma> two sites can host the same material, too
<arma> in fact, i recommend that.
<arma> that way if one of them goes down, the other one is still up. i would recommend that all of the backbone sites contain all of the info.
<arma> and there be multiple search sites.
<rhlufaq> Definately.  Problem is that you get 10 sites around the world willing to mirror networking info, and only 2 sites in Austria willing to do file-systems.
<rhlufaq> You don't get equal distribution- just like in DNS (which is slow)
<rhlufaq> Don't get me wrong- I think for news, this is a great way to go.
<rhlufaq> I'm not sure this scales well for a KB though.
<arma> sure. you don't need equal distribution.
<arma> the internet is fast.
<arma> you just want redundancy.
<rhlufaq> The internet is fast?  What internet are you talking about?????  (Took me 3.5 hours to download StarOffice 5.0 over a T1)
<arma> you weren't using most of the T1, then.
<arma> I'm on a T1 here, and I can transfer a gig in perhaps 10 minutes.
<arma> err, that's off by an order of magnitude, sometimes.
<rhlufaq> Exactly.  I have fast connectivity, but I can't see that across the internet- only to my POP in most cases.
<arma> (that is, better.)
<rhlufaq> Just recentely I've seen 2500ms latency from San Jose, CA to Seatle, Washington .
<arma> that's because you're using bad networks. in general, the internet can handle load. furthermore,
<arma> this is actually relatively light load -- the queries and responses don't take much "space"
<arma> the only real issue will be propagating the data. and new data won't be coming in quickly enough for that to matter.
<arma> now, it could be a problem if you wanted to create an index off the remote data nightly. that would begin to suck quickly.
<arma> a better algorithm would use an incremental indexing system, so you only sent it your changes for that day.
<arma> i'm sure such algorithms exist.
<rhlufaq> They do.
<rhlufaq> [thinking...]
<arma> it's not like we're providing linux distribution sites, where a lot of data will be changing hands. if we do this right, it will be distributed such that no server spends much cpu time.
<arma> we don't need massive machines. we just need a lot of volunteers.
<rhlufaq> good volunteers
<arma> no, just some of them have to be good.
<arma> only a few, really.
<arma> the rest of them can spend most of their time off the net, it doesn't matter.
<rhlufaq> Maybe I just have issues since I'm a control freak when it comes to computer systems.
<arma> i'm happy to continue debating this. this is a pretty generic distributed system. it's extremely powerful and flexible.
<arma> i can also answer questions, etc, rather than making it a debate. :)
<arma> i'm a control freak too. that's why i have so much redundancy in the design.
<arma> in your design, if your machine goes off the net, you lose. the whole site is down.
<rhlufaq> Pro's & Con's.
<rhlufaq> The users who bookmarked that machine think the site is down, but everyone else see's it as up.
<arma> that's where it gets tricky --
<rhlufaq> If you move the redundancy off of the DB protocol and make the machines redundant, then you can loose one (N+1) and everyone sees you as up.
<rhlufaq> Both have pro's & cons
<arma> you can register an address as a set of IPs -- that's how mx and dns work --
<arma> such that it picks a machine that's up. i don't know the full details of using it for web, though.
<rhlufaq> You can't do it with a web server though... you end up with RR DNS which is load balancing not, redudancy
<arma> so in reality, www.linuxkb.org would yield (in round robin) www.aa.linuxkb.org, www.ab.linuxkb.org, etc.
<rhlufaq> You need a hardware or special software solution.
<rhlufaq> Right, so if you have 2 machines and 1 goes down, 50% of the people think you're dead.
<arma> in any case, that is a stumbling block currently, yes. i haven't solved it, but i think it's acceptably solvable.
<arma> sure. or try again.
<arma> that's not very good tho, i agree. it will confuse people.
<rhlufaq> Nope- your client caches the IP DNS tells them for the host
<arma> hrm. true.
<rhlufaq> And many DNS servers ignore the TTL on DNS entries if you set it too low.
<arma> so i guess if you had a site that was up most of the time,
<arma> then it would keep track of the servers that were up, and it would 302 you at them
<rhlufaq> There are comercial solutions available. 
<arma> <META NAME="refresh" content="2; url=http://www.seul.org/">
<rhlufaq> There's also a DNS server which acts like you just said.
<arma> yeah. there are solutions, and they're not commercial.
<rhlufaq> A HTTP Redirect though, changes the URL in the client browser so they bookmark a specific machine
<rhlufaq> not the www.linuxkb.org 
<arma> i think that's along the lines of a good solution, tho
<arma> if you frame it, the url doesn't change. but that solution sucks.
<rhlufaq> This like you said isn't insurmountable... I've done it at my current company, and NaviSite has the comercial solution available to us should we ever need it.
<rhlufaq> [gawd that was a bad sentance]
<arma> it wasn't that bad.
<arma> ok. any other holes you see? :) are you convinced that this is worthwhile for longterm?
<rhlufaq> Since Linux is a global OS, yes it is worthwhile.  I strongly disagree with the notion only a few volunteers need to be "good", but only time will prove one of us correct.
<arma> well, a site which is consistently down will not get handed many clients
<arma> so it will start getting less use.
<arma> and this really doesn't have to be a 'global' network. it can really be just 3 or 4 machines, all located in california or whatever.
<rhlufaq> But one that is not up to date will continue to get people who get fustrated that the info is poor.
<arma> the protocol is extensible in a bit way.
<arma> er, big way.
<rhlufaq> Admins will screw things up- run out of disk space, turn off the update daemon, etc
<arma> yeah..that's one of the big problems i have currently, actually.
<rhlufaq> space or update?
<arma> the current propagation (transport) mechanism is nntp. a good nntp server (eg INN, CNEWS) is complex and difficult to maintain.
<rhlufaq> got that right.
<rhlufaq> Just package it into a RPM
<arma> yep. know any inn gurus?
<arma> neither do i
<rhlufaq> Ask Slashdot
<arma> ung.
<arma> you're kidding.
<rhlufaq> No, use the "ask slashdot"
<rhlufaq> Have them post a request for you.  Or LWN for that matter
<rhlufaq> Still though, nntp is a pain in the butt to install and admin.  Lottsa issues- you're going to scare away a lot of possible sites
<arma> it's true.
<arma> lwn is a good option. i'll consider that one further.
<arma> i think in a couple weeks, it could be reasonable.
<arma> basically, if i can get somebody to package inn so it's configured to do the right thing,
<arma> then what remains is some scripts to take the spool and convert it into news in the right format for that site,
<arma> and some scripts to convert news into a spool article. and inn would take care of propagating it.
<arma> i was actually looking into using some cheaper programs (eg leafnode, pnews (not Pnews)),
<arma> because they're shorter and easier to understand. but last i checked, i needed most of the features of inn to do this right.
<rhlufaq> Such as?
<arma> (cnews is out of the picture. inn is better, from what i can tell.)
<arma> such as..i wanted authentication.
<arma> it's been several months since i thought about this. hrm.
<rhlufaq> So each server has to be configured where to get each "topic" right?
<arma> http://www.seul.org/archives/lu/news/Aug-1998/msg00034.html
<arma> yep
<arma> i posted to news.software.nntp a while ago,
<arma> and their response was pretty uniformly "get off your ass and learn inn"
<rhlufaq> Nice fellows
<rhlufaq> Excuse this question, but why not http?  Has authentication, easy to setup, configure and maintain.  Just run the job via cron.
<rhlufaq> push vs pull...
<rhlufaq> Probably not so good for news, but very usable for KB
<arma> hm. well, a framework is needed to keep track of unsent messages, deal with retries, etc
<arma> inn already is designed to do all that.
<arma> it could be done via another protocol. but the nntp protocol (there's a link to it from the lu page) is designed exactly for this.
<arma> but it's true that speed isn't as much of an issue for the kb
<arma> just about any protocol would work.
<arma> a bad one will load your machine down more, but nobody will notice anyway, most likely.
<rhlufaq> I think that's the issue I see- nntp is excellent for news for those exact reasons. But a KB doesn't need that.  This added level of complexity that nntp adds kinda scares me.
<arma> plus, port 139 is known for being a security problem.
<rhlufaq> Ok, change the subject...
<rhlufaq> So a server knows it can't fullfil a request- how does it make a request to the server that does?
<rhlufaq> (for the client)
<rhlufaq> Also what about the caching mechinism that Dana talked about?
<arma> option one: it makes the request of the server, and answers the client.
<arma> in this case, it checks its cache before doing it, and if it's in the cache, returns that,
<arma> else caches the response so it'll have it for next time.
<arma> option two: tell the client where to find it, and he clicks on the new url.
<rhlufaq> I knew that :-)
<rhlufaq> What I'm asking is what kind of behind the scenes protocol are you talking about?
<arma> a really thin one.
<arma> it doesn't have to do much.
<arma> it will be a bit of skeleton built around the webserver
<rhlufaq> Is the plan to have it make queries to the SQL/whatever backend DB?
<arma> basically, each server is going to be dynamically generating its responses, right?
<arma> then it can plug in the appropriate url in that response,
<arma> either a cached entry or the real thing.
<arma> alternatively, it could always use a local url,
<arma> and that local url would either forward you or return the cached entry
<arma> this would confuse people who bookmark fanatically, though.
<rhlufaq> Actually the idea was to nightly dump the SQL DB to web pages via a template and serve static pages
<arma> not for search engines.
<arma> (i hope)
<arma> search engines need to be able to dynamically generate html,
<arma> since they don't know what the queries will be.
<arma> hrmm.
<rhlufaq> You'd have a different DB to act as the search engine.  SQL is a poor performer for this type of thing (especially fuzzy logic) from what I hear
<arma> but the question is, what url does a knowledge server use, if it knows the knowledge is on a different server. hrm.
<rhlufaq> Frames
<arma> well, that is answered based on how we solve the one-name-for-many-machines problem.
<rhlufaq> I think you basically have to make it look like the response came from the "local" server
<rhlufaq> You not only have to distribute the SQL DB but the search engine DB too
<rhlufaq> Yuck yuck yuck
<arma> no, wait..
<rhlufaq> Make it socket based, and redundant in the calling app
<arma> the search engine db needs to be on a different machine from the knowledge servers anyway, right?
<rhlufaq> No.
<arma> well, ought to be?
<rhlufaq> Faster indexing if on local machine
<arma> what if it's incremental indexing, as we discussed before?
<rhlufaq> But sitll, it should be a socket call.
<rhlufaq> s/sitll/still./
<arma> a socket call to do what?
<rhlufaq> Queries
<rhlufaq> This way you can failover internally and externally without the user being the wiser
<arma> err
<arma> search requests or knowledge requests?
<rhlufaq> KB
<rhlufaq> Search engines you can build backup servers into the form
<arma> from the search engine?
<rhlufaq> lost me
<arma> who does the socket stuff?
<rhlufaq> Your CGI script serving the user request for a KB document
<arma> where does this cgi script run?
<rhlufaq> It via a socket call requests the document from the local machine, and fails over to remote machines
<arma> ok. so that means that all linuxkb servers act like they have all the data.
<rhlufaq> CGI runs from any server you want it too
<rhlufaq> Right
<arma> including for searches, etc?
<rhlufaq> Not sure... that's different.  The search engine will likely be built off an existing web search engine.  You'll have to front end it if you want to make it a socket call.
<arma> (which we can do)
<rhlufaq> But I don't think that is necessary.  Just build into the form the ability for the user to query different search engines (which all have all the info)
<rhlufaq> If you front end it, it gets really complicated, because it has to query *all* the servers and reformat the responses into one page
<rhlufaq> The search engine doesn't have the recursion that we can build into the KB query protocol.
<rhlufaq> Or am I on dope?
<arma> sure
<rhlufaq> Thanks. :-)
<arma> well, i'm still a bit fuzzy on a good way to do the searches, assuming the search engine might not be on a knowledge server
<arma> but i think it's solvable. easily solvable if we don't pick the absolute best method.
<arma> ok. problem one:
<rhlufaq> Well the KB DB dumps its data into static html pages, which the local/remote search engine can then index.
<arma> what if the knowledge server you request is down? then you need to get punted to a different one.
<rhlufaq> Cisco Distributed Director
<arma> you say the search engines give you a list of knowledge servers which will serve your query,
<rhlufaq> NaviSite has a few which we can get access to I believe
<arma> and if one is down you pick the next one on the list? i'd prefer to have it more automated than that
<rhlufaq> Actually I (meant to) say that the knowledge servers would give you a list of search engines to use.
<arma> so how do you pick a different KS if the one you've bookmarked is down?
<rhlufaq> If you want it automated, you have to front end the search engines with another socket app
<rhlufaq> Hold on...
<arma> ok. here's another thought to throw into the fray: squid. a caching web proxy. if we're making these connections via http, then squid is awfully good at caching stuff.
<arma> (efficiently)
<rhlufaq> Ok... I'm back
<rhlufaq> Squid? I'm familar with it, but not in this way....
<rhlufaq> I've got a solution for the KS redundancy that will also work for the search engine
<rhlufaq> What you do is each server runs a local DNS server.  All DNS servers only have their IP for each host name.  All DNS servers are listed as primary/secondaries...
<rhlufaq> When a server goes down, it can no longer serve it's IP as a web server.  Not perfect, but free
<arma> er
<rhlufaq> There's also a Perl DNS server which can do this sorta thing
<rhlufaq> But I can't find the link
<arma> err, that's ignoring caching and propagation effects.
<rhlufaq> I thought I read about something simular, but I can't find it either at the moment.
<rhlufaq> Right- the cache screws ya, so you set a low TTL that some caches ignore
<arma> that's an ugly solution. there exists an elegant one.
<rhlufaq> Your only other option is something like a Cisco LD/DD combo, Resonate, or RND Networks WSD-DS
<arma> ok. here's the deal. you request a doc from a KS. it doesn't store that document locally, so it requests it from its local proxy server because it knows which server has it.
<rhlufaq> Unless you have all your servers on a local LAN, in which case they can masqurade for each other when one goes down
<arma> that proxy server (squid) either fetches it or already has it cached.
<arma> somewhere in there, that KS also keeps a list of which servers are down and which are up (since sometimes its requests fail, it knows)
<rhlufaq>  thought we were trying to figure out how a client get's to the KS in the first place when it is down?
<arma> i'm just trying to think through things for now. :) to make sure
<arma> so this works for search engines as well. specifically, they can run their own squid, and answer queries that would normally go to KS's
<arma> that seems like a waste, though, since that loads the searcher more.
<arma> but anyway, it seems like the search engines are going to be the ones that keep track of which KS's are up, and the ones that "interface" with the KS's
<arma> meaning most people will have a searcher bookmarked, not a KS. right?
<rhlufaq> I don't think squid is buying us anything- it's pretty disk/memory inefficent
<arma> ok. not squid then. but a caching system of some sort.
<rhlufaq> People book mark the KS (SQL DB) which has the index for browsing.  The SS is a background app (glimpse, ht://dig)
<arma> ok. then i'm confused. i thought the SS would potentially be on a different machine, and people would be mostly quering the KB through searches
<rhlufaq> The SS can/may be on a different machine.  I see a search engine as a backend app rather than a front end.  The search form would be on the KS....
<arma> if the search form is on the KS, then where does the search server come in?
<rhlufaq> It would call a CGI (local) which would open a socket to a SS (local or remote)
<arma> eek
<arma> why not go to the SS directly?
<arma> i was treating the KS as a backend
<rhlufaq> I get it- make the KS totally invisible to the user
<rhlufaq> Make is search only?  No browsing?
<rhlufaq> Nevermind.
<arma> isn't browsing similar to search?
<arma> you certainly use an index for it..
<rhlufaq> no.  Browsing implies a human readable index vs one optimized for a computer.
<rhlufaq> Kinda like a file system tree.
<arma> the SS could have both
<arma> i guess i'm assuming that in a lot of cases the SS and KS will be the same machine, tho i'm allowing for them to be different machines.
<rhlufaq> Right.  The SS index and the KS index are generated from different sources though.
<rhlufaq> The KS index is generated from a SQL DB, and the SS index is generated from the actual KB
<rhlufaq> which is a part of the KS
<rhlufaq> scratch that!
<rhlufaq> The KB is also generated from the SQL DB.
<rhlufaq> Yeah.
<arma> try that again? i think the acronyms got confused
<rhlufaq> Step 1:  KB data is imported into a SQL DB
<rhlufaq> Step 2: SQL DB is exported into HTML files (via templates) and a KB is generated (with a human browsable index)
<rhlufaq> Step 3: A search engine indexing program is ran against the HTML of the KB which creates a search engine DB (DB2 in the case of ht://dig)
<rhlufaq> ---
<rhlufaq> Each step can be on the same or different machines.
<rhlufaq> You alive?
<arma> http://www.cflash.com/~yury/index2.html
<arma> just noticed 'newsbot' on freshmeat, went to find it.
<arma> he's got some scripts that parse slashdot/freshmeat so you don't have to go there
<arma> ok. reading..
<arma> (he also appears to be a full-fledged WareZ D00D, but hey..)
<arma> ok. i agree with your assessment of steps 1-3
<arma> i think that there are two primary interactive things we're doing here:
<arma> we're having people do searches, and we're providing the actual pages.
<rhlufaq> I was planning on doing a simular thing for our linux KB- a "what's new in the world" section.
<arma> i vote steps 1 and 2 happen on the same machine, and that machine serves the static html pages for the docs themselves.
<rhlufaq> That makes sense.
<arma> so what can be done now that's useful for both short-term and long-term?
<arma> people can work on the tables for the knowledge itself,
<arma> people can figure out how to build the static pages,
<arma> people can set up a searcher that indexes those pages,
<arma> and then for short-term we can connect them together on a single machine and run it.
<arma> longer-term, we can connect them with a bit more complex glue, including the cgi to deal with caching or remote fetching, etc.
<arma> sounds like a pretty straightforward plan.
<rhlufaq> Sounds good to me too.

PGP signature