Re: How to manage mirrored content

[Wade Hampton <whampton@staffnet.com>]

Aaron Turner wrote:
> 
> Mostly to Wade, but interesting to the rest of the world too.
> 
> Ok, I think I figured out part of the problem.  A big part of the problem
> is the lack of tools (or even what these tools should be) to manage the
> 1000's of documents in /mirrors, such as man pages and whatnot.
> 
> There are a number of obsticles to overcome:
> 
> 1) Creating a elegant tree based categorization structure where things are
>         put where they make the most sense.
> 2) Determining where each doc should exist in that tree
> 3) Actually having the tools to initially put in and then manage the docs
>         in the tree.
> 
> This idea mostly deals with #3, but it makes #1 and #3 easier.
> 
> I initially just tried reading articles in the RHLUFAQ and then creating
> the cats manually in the DB and then manually interting the articles.
> Time consuming, slow, and difficult to correct errors.  We just don't have
> the tools to effectively manage the data in this way.
I agree.
> 
> So I started trying to find tools to help me represent the tree structure
> so that I would make less mistakes and do things faster.  Things like Star
> Office, Visio, and things like that.  Problem is that all these tools got
> in my way.  Then it hit me- what existing tool is natively tree based,
> supports links and has 100's of different tools to manage it?  Why a Unix
> filesystem of course!
> 
> So here's my idea:
> 
> Convert the docs to HTML where approprate (man pages for example).  Then
> in the filesystem create the cat tree and place the docs in the tree where
> they belong.  Use mc, tree, ln and whatever to make this easy.  Then
> someone (me or Wade) can write a set of perl scripts to import/export this
> structure into and outof the database.
This was the approach that I was starting.  For example, converting
GNU info pages to html with info2www then mirroring the local WWW 
server to a directory -- works well (prototyped this weekend).
> 
> There are a few limits with this idea, but are solved with eating more
> disk space.  For one, manpages link to other manpages.  Initially it will
> be difficult to have the html point to the correct location inside the DB
> tree structure, so we need to have a copy of the man pages in their
> standard flat structure.  This allows tools like man2html to do their job.
> No big deal since we've got 45GB in /home :)
1. convert man, info, etc. to html
2. put them in directories similar to the original man/man1, man/man2...
3. if there are link problems in the html (references to man2html)
   change man2html to link to the file, not calling man2html to 
   get the next man page (should be rather easy me thinks).
> 
> After listening to Wade, I'm also under the impression that the only way
> we're going to get man pages and other mirrored content into the DB is
> boring labor.  I don't believe that a tool will be effective to drag in
> large amounts of data accurately- especially into a tree without any
> existing structure.  To create that kind of AI is very difficult because
> keyword matching isn't enough.
Much will be manual, but we should be able to come up with some scripts
to automate some of it -- e.g., build a page with links to items 
relating to the concept (e.g., ppp, apache, etc.).
> 
> There's also the problem of keeping changes different people do insync
> with others.  Usually this is called CVS, but you don't want to use CVS to
> manage file locations because it is a tedious and bandwidth intensive
> process in such a situation.  Hence here's my idea around this:
> 
> Create a CVS module with one file in it, the output of the tree command
> against that directory structure.  This is compact enough to not require
> much bandwidth and will actually use the strength of CVS (change control)
> to our advantage.  You will then be able to edit the tree file
> (cut-copy-paste) or the actual directory structure and then commit the
> changes.  Then on the server the webupdate script will process the tree
> file and create the directory strucuture which another script will then
> import into the DB.  This has the advantage of making it complicated on
> the backend thereby making it simple for the enduser (the poor slob doing
> the categorization).
Not a bad idea.  We will need the tools on the backend to automatically
generate the tree, however (see comments above).
> 
> We could theoretically skip the process of creating a real directory
> structure and go straight from tree output format to the DB, but I think
> the ability to use tools like Midnight Commander makes it worthwhile to do.
Agree.
> 
> The one thing that we don't have currently that we will need, but is easy
> to do is a specific DB type to discriminate local external content from
> remote external content.  Otherwise you have to use the URL to determine
> this and that really isn't the correct/safe way of doing that which may
> lead to the DB being filled with bad info.
Yep -- that would be a good idea and save some time/WWW accesses.

Also, we will need a "check" tool to check the tree versus the database
and report if any files or links are missing.  
> 
> Lastly this tool would be an excellent basis for another tool we will need
> in the future- a LKBP export tool so that you can put all the content of
> the KB on a CDROM without having to go through the trouble of a SQL DB,
> PHP, ModPerl, etc.
I like this idea VERY much -- it would be like some CD's I have seen
(I think Sun had such a while back).  

I have been using man2html for a while, but the current version 
just converters a single man page.  I need to play with it a bit.
I still think using it to convert the man pages to html, then fixing
the links might be a good idea.  

I'll look at the manual page issue tonight.  I think the info 
stuff is converted (but 2K files in a single directory -- ouch).

Cheers,
-- 
W. Wade, Hampton  <whampton@staffnet.com>  

Linux is new, it's come out of nowhere, it's free. Hard to beat that 
price. And it's taking marketshare left and right.  

--Microsoft's lawyer, in closing argument, Bristol v. Microsoft,
http://www.microsoft.com/presspass/bti/07-15closing.htm