Re: control freak? Maybe.

[Aaron Turner <aturner@linuxkb.org>]

First, I'm happy to say that Victor has accepted our offer to work on the
external mirroring system.  I've already sent in a subcribe request for
him to the list.  (Victor, sorry if you receive a dup on this, but I
figured that would be better than zero copies)

Now on to your comments...

On Wed, 7 Jul 1999, Jason Pincin wrote:

> External documentation will be integrated into our database structure
> and will be part of the tree.  I'm proceeding down this road carefully
> though so as to fine tune it enough that external documentation
> doesn't overrun all native articles.  But yes, is the answer to your
> question, it will be.  I'd like an option in the search engine to
> filter it out, but I'm not sure if you can provide that with ht://dig
> or not?

Sounds like we're on the same page then.  There are a lot of questions
we'll have to deal with about how to keep our content up to date but still
maintain quality control.  This won't be so important with the LDP or
Linux Gazette which have their own quality control measures, but if we end
up mirroring smaller sites this will become an issue.

With regards to your ht://dig question, I'm not sure.  Basically, you're
asking to do an "exclude hit by URL" where if the URL matches some
substring we don't count it as a hit.  Before I go on, let me digress a
bit.

When we put copywrited material (or just link to it) from other sites into
our MySQL database, we need to be very careful.  Our license as it reads
now, says that content off of /mirrors/ is copywrited/licensed by the
original author.  This is good for us because it makes dealing with other
peoples' licenses simple.  It's good for the community, because it's
simple to intepret.  It's bad for us, because it makes integration between
the MySQL db and the mirrors complicated without it looking like we're
changing the copywrite/license of mirrored content.

Let me explain...

When a user is browsing our category tree or the related links section of
an article, and clicks on a link to mirrored content, how is that URL
presented to the user?  My guess is that the easiest thing to do is:

/mirrors/LDP/article.html

But that doesn't have any category information in the URL
which breaks our drill-down feature.  If we re-write the URL 
like:

/mirrors/LDP/article.html?&c=1.4.6.8.10

(which is valid)

This allows us to categorize content in our native categories without
putting content like the LDP under the umbrella of our license.  The next
best thing is:

/articles/mirrors.php3?id=7&c=1.4.6.8.10

and have mirrors.php3 read the .html file associated with id=7 in the
MySQL db and append a license disclaimer to the end of the page.  This is
kinda ugly but it does have a really nice side-effect which I'll explain a
bit later.

The point is that for the mirrored content to be drill-downable, it *must*
have category information (c=a.b.c.d) in the URL.  Without it, you can't
drill-down search mirrored content.

Now, back to your question of limiting search results by mirrored and/or
unique content.  To do this, almost positively requires the second method
of accessing mirrored content (via a PHP front end).  The reason is that I
need a *static* substring to match against to differenciate between
mirrored and non-mirrored content.  Basically URL's have to be written
like this:

/articles/article.php3?c=a.b.c.d&id=xxx
/articles/mirror.php3?c=a.b.c.d&id=xxx

The order of the CGI params is IMPORTANT.  I can only search *one* string;
hence the content type (denoted by the name of the PHP page) and the
category identifier string (c=a.b.c.d) must be contiguous.  Hence to
filter out all mirrored content, I limit the search to URL's matching:

article.php3?c=a.b.c.d

to filter out unique content:

mirror.php3?c=a.b.c.d

to include both mirrored and unique content:

?c=a.b.c.d

> As for the form for mirroring, to make everything nice & automagic,
> it's a good idea, but again, we'll have to impliment it carefully.  
> We don't want it abused.  Again, we'll probably integrate that into

Agreed, no need to mirror warez (or worse).

> the moderator system, have the form submit to the moderator form.  
> Upon moderator mirror approval, the scripts are put in place and it
> begins.  The moderators can also approve it's integration into the
> tree.

Agreed.

> As for these people maintaining the email gateway as well, there some
> options there.  I think we need to discuss EXACTLY what each of us see
> that gateway doing and the process it will go through before any
> decisions are made there.  Some questions I have for you are:
> 
> Will the email gateway allow only pre-formatted messages or free text?

Both.

> How do you see it fitting into the moderator system?

Basically we have two classes of email submissions:
1) Those who are already a moderator/author listed in our user database
2) Those that aren't.

Those that aren't, get their email put into the generic moderator database
queue.  Those that are get it automattically assigned to them (per email
address).  We give the email a TTL of say 5 days.  If the TTL expires
while the email is in the person's queue it gets moved into the general
delivery queue.

> Etc.

Yes, I prefer 'etc' over 'yadda-yadda-yadda'.  I also prefer 'ex' over
'eg'.  :)

> I have my own ideas and views, but lets talk about it.
> 
> I see it growing immensly though.  Here's how I ultimately see the
> email gateway working (not only do we become THE support portal, but
> we become THE support forum):
> 
> Email gateway will take either free text or formatted text. If
> formatted text is given, the article is generated and posted for
> moderator approval. I'll explain free text in a few lines.
> 
> A 2nd part to the mail gateway enables something I've seen from the
> beginning (not needed for go-live either).  The ability for someone to
> mail a question from the site using a web-form.  To summarize, the
> form directs the questions to the proper people by information already
> built into the DB.  People that sign-up as KB support Reps (or
> whatever) fine tune how many mails they wish to get per day or week,
> and what category of questions they wish to recieve.  The mail is
> recieved, and the interchange of mails begin betwen the people.  The
> reply-to's, etc are set up in such a way that the KB tracks all these
> mails until case closure, and then builds a transaction log for the
> conversation which gets posted in a forum for volunteers to turn into
> an article.

Hmmmm... this is a unique way of going about things.  It probably should
solve the issue of off topic posts on maillist/bulletin-board centric
support methodologies.  One thing we'd have to have is a way for a rep to
say "I don't have a clue" and pass it on to others.  Not difficult though.

This reminds me a lot of that pay-for Linux support site I mentioned last
month. The one where you ask a question and say how much you're willing to
pay.  The question then get's passed to a person who answers it or asks
for more info.  The Q/A ends up in a database that anyone (including us)
can access.

> Now, from above, the free-text submision into the mail gateway would
> also be thrown into this forum.

OK.

> This mail system I describe along with the mail gateways, etc. would
> allow the KB to really become the ultimate support mailing list.  
> More configurable than any other list out there.

I think this is on the right track.  Free-for-all maillists/bulletin
boards are becoming off-topic flame-fests which is driving the
knowledgeable people off.  Case 'n point: Ramon Gandia's recent leaving of
the RedHat List.  He's not the only one to leave either.  I haven't been
active on the list for almost a year now, because I got sick of all the
non-Linux/support questions.

> Thoughts are welcome.

I think it would be wise to bring on another two people to start the email
gateway.  We mentioned somone in our chat yesterday: Norbert Kuemin.  He
should be good on the admin/Rep pages for the front end, but we're going
to have to find someone with better Perl and some SQL experiance for the
backend.  James Fu would be a good compliment (but we need determine the
effect of his employment at Linux Care will have).  Other people that
catch my are are Jakub Linowski, Charles Short, and Francisco Sam
Castillo.

--
Aaron Turner, Core Developer       http://vodka.linuxkb.org/~aturner/
Linux Knowledge Base Organization  http://linuxkb.org/
Hanlon's Razor:
Never attribute to malice, that which is adequately explained by stupidity.
 
> Jason
> 
> 
> On Tue, Jul 06, 1999 at 01:13:29AM -0700, Aaron Turner wrote:
> > 
> > Well after reading 50+ submissions I'm not sure I want to share the search
> > engine.  Last I heard from Jason, the hacks to htsearch won't be necessary
> > with the new database structure.  Assuming that's true, it would be faster
> > for me to make the final last modifications to the search engine myself
> > than try to get someone else up to speed.  If the hacks are still
> > necessary, then maybe one of:
> > 
> > Thomas Buschhorn
> > Victor Stanescu
> > Dean Giberson
> > 
> > would be helpful.
> > 
> > For the LBF, since this isn't necessary for go-live, no need to deal with
> > trying to manage people working on this project.  I'm not sure how big of
> > a project this is right now since I'm still working on the feature list.
> > 
> > The mirror project is going to require some complex Perl scripts and some
> > SQL knowledge.  I found a few people who I think would be a good match,
> > but honestly, before I bring anyone on, Jason and I need to come to an
> > understanding on what this is for.  My goal for the record is to (not sure
> > how to do this) integrate content that isn't written specifically for our
> > site (LDP, Linux Gazette, man pages, etc) into the database.  The issue is
> > that we have to present the external content in as a URL in the form of:
> > 
> > http://www.linuxkb.org/mirrors/
> > 
> > Hence, external docs will not only be listed in "related links" but in the
> > actual category tree.  Also, I expect the email gateway we discussed will
> > maintained by this project/group as well.
> > 
> > One thing we can start immediately and that anyone can do is start
> > tracking down good sites to mirror.  Anyone who doesn't have the
> > experiance or time to do development can scour the web for sites we should
> > mirror.  
> > 
> > Another bright idea I have is an auto-mirror form.  Basically a site admin
> > goes to this form, fills it out, and it sets up the scripts to start
> > mirroring their site.  As an added bonus, they have the option of checking
> > a box signaling they grant us the right to include their content in our
> > database.  This makes the legal chore of contacting each site admin way
> > too simple.  :-)  We could announce this service on Freshmeat/LT/etc and
> > I'm sure we'll see scores of small Linux support web sites sign up.
> > 
> > Basically, we become *the* Linux support _portal_.  The possibilities are
> > endless.
> > 
> > --
> > Aaron Turner, Core Developer       http://vodka.linuxkb.org/~aturner/
> > Linux Knowledge Base Organization  http://linuxkb.org/
> > Hanlon's Razor:
> > Never attribute to malice, that which is adequately explained by stupidity.
> > 
> > 
> > 
> > 
> > 
> > 
> > 
>