How to manage mirrored content

[Aaron Turner <aturner@linuxkb.org>]

Mostly to Wade, but interesting to the rest of the world too.

Ok, I think I figured out part of the problem.  A big part of the problem
is the lack of tools (or even what these tools should be) to manage the
1000's of documents in /mirrors, such as man pages and whatnot.

There are a number of obsticles to overcome:

1) Creating a elegant tree based categorization structure where things are
	put where they make the most sense.
2) Determining where each doc should exist in that tree
3) Actually having the tools to initially put in and then manage the docs
	in the tree. 

This idea mostly deals with #3, but it makes #1 and #3 easier.

I initially just tried reading articles in the RHLUFAQ and then creating
the cats manually in the DB and then manually interting the articles.
Time consuming, slow, and difficult to correct errors.  We just don't have
the tools to effectively manage the data in this way.

So I started trying to find tools to help me represent the tree structure
so that I would make less mistakes and do things faster.  Things like Star
Office, Visio, and things like that.  Problem is that all these tools got
in my way.  Then it hit me- what existing tool is natively tree based,
supports links and has 100's of different tools to manage it?  Why a Unix
filesystem of course!

So here's my idea:

Convert the docs to HTML where approprate (man pages for example).  Then
in the filesystem create the cat tree and place the docs in the tree where
they belong.  Use mc, tree, ln and whatever to make this easy.  Then
someone (me or Wade) can write a set of perl scripts to import/export this
structure into and outof the database.

There are a few limits with this idea, but are solved with eating more
disk space.  For one, manpages link to other manpages.  Initially it will
be difficult to have the html point to the correct location inside the DB
tree structure, so we need to have a copy of the man pages in their
standard flat structure.  This allows tools like man2html to do their job.
No big deal since we've got 45GB in /home :)

After listening to Wade, I'm also under the impression that the only way
we're going to get man pages and other mirrored content into the DB is
boring labor.  I don't believe that a tool will be effective to drag in
large amounts of data accurately- especially into a tree without any
existing structure.  To create that kind of AI is very difficult because
keyword matching isn't enough.

There's also the problem of keeping changes different people do insync
with others.  Usually this is called CVS, but you don't want to use CVS to
manage file locations because it is a tedious and bandwidth intensive
process in such a situation.  Hence here's my idea around this:

Create a CVS module with one file in it, the output of the tree command
against that directory structure.  This is compact enough to not require
much bandwidth and will actually use the strength of CVS (change control)
to our advantage.  You will then be able to edit the tree file
(cut-copy-paste) or the actual directory structure and then commit the
changes.  Then on the server the webupdate script will process the tree
file and create the directory strucuture which another script will then
import into the DB.  This has the advantage of making it complicated on
the backend thereby making it simple for the enduser (the poor slob doing
the categorization).

We could theoretically skip the process of creating a real directory
structure and go straight from tree output format to the DB, but I think
the ability to use tools like Midnight Commander makes it worthwhile to do.

The one thing that we don't have currently that we will need, but is easy
to do is a specific DB type to discriminate local external content from
remote external content.  Otherwise you have to use the URL to determine
this and that really isn't the correct/safe way of doing that which may
lead to the DB being filled with bad info.

Lastly this tool would be an excellent basis for another tool we will need
in the future- a LKBP export tool so that you can put all the content of
the KB on a CDROM without having to go through the trouble of a SQL DB,
PHP, ModPerl, etc.

Comments?

--
Aaron Turner, Core Developer       http://vodka.linuxkb.org/~aturner/
Linux Knowledge Base Organization  http://linuxkb.org/
Because world domination requires quality open documentation.
aka: aturner@vicinity.com, aturner@pobox.com, ion_beam_head@ashtech.net
The difference between `Unstable' and `Usable' is only two characters: NT