Re: 1 major problem

[Aaron Turner <aturner@linuxkb.org>]

Recap:

Jason and I talked on Sunday via IRC about the search engine and what
htdig can and can not do.  (I'll post the log of the IRC soon, which will
make things more understandable below.) One of the big issues was that
Jason wanted to be able to restrict the results of a search by the URL.  
While htdig can do this, it can currently only match on one single string.

This means that you can't say:

restrict1=path1&restrict2=path2

or

restrict=/c=1\.3\.\d/

Basically what Jason wanted to do was in the case of:

c=1.2.5.9
c=1.2.5.6
c=1.2.5.20

to be able to tell htdig to only search two of those paths without the
3rd.  I thought I could hack some perl to remove matches that we didn't
want.  Like if we wanted 1.2.5.9 and 1.2.5.20 but not 1.2.5.6, we'd
search for 1.2.5 and remove all matches containing 1.2.5.6 post-htdig.

The problem is that for searches that return a lot of results, those
results are likely to span multiple pages of results.  What happens is
that htdig will return X results per page.  So if htdig returns 25
results, and 1/2 of the hits are in 1.2.5.6 the page will say:

Search results 1 - 25 of 72:

But only have 13 results in the page.  This obviously is confusing
to the user, and they're likely to think the search is broke.

I read on the htdig list archives, that they're working on a regexp
enabled restrict paramter.  Ie you could say:

restrict=/(1\.2\.5\.9|1\.2\.5\.20)/

Looks like the regexp feature is complete in the 3.2 development tree
(we're at 3.1.2 which is the latest stable).  Geoff isn't sure if he's
going to back port it right now.  I'm going to ask around about what
experiance people are having with 3.2 and if it's production ready or not.
(My guess is no.)

Also, I realized that this problem is bigger than I first thought.  It not
only happens when you use the Application option (as Jason and I discussed
using it), but any time you do a search from high in the tree (such as
c=1) and htdig finds duplicates because the same article lives in two
different places in the tree. (Our goal is to remove the duplicates.)

There is only one way to really fix this problem:

Make sure that duplicate articles have the same URL.  This means we either
strip the c= param from the URL or set c= to be the same for all cases.
Ie:

/articles/article.php3?id=10&c=1.3.5.9 becomes:
/articles/article.php3?id=10

I'm sure everyone sees the problem with this- you can't use the URL to
drill down searches in htdig anymore.

Basically, we're screwed either way, and I don't know how to work around
it.  I'm going to email the htdig list and see if anyone has a solution.

-Aaron


On Mon, 14 Jun 1999, Jason Pincin wrote:

> Bummer... but what can ya do?  :)  Let me know what kind of response
> you get from the htdig guys.
> 
> On Sun, Jun 13, 1999 at 09:28:43PM -0700, Aaron Turner wrote:
> > 
> > Ok, so there's one "major" problem with this whole post-htsesarch restrict
> > idea.  Basically it will happen when many results are returned- more
> > results than would fit on one web page.
> > 
> > Basically since we will be removing hits after htsearch has generated
> > them, htsearch's understanding of how many hits there are will be
> > incorrect.  Example:
> > 
> > Joe drills down to 1.3, selects Apache, and types in "performance" as his
> > keyword.  htsearch finds 35 hits, the first 25 of which will appear on
> > the first page, the remaining 10 on the second page.
> > 
> > Problem is that my mod_perl script will remove hits after htsearch figures
> > out what hit belongs on what page.  So while the first page says:
> > 
> > Hits 1 - 25 out of 35
> > 
> > Maybe 1/2 of them are duplicates.  This means the user expects 25 hits on
> > the page, but only sees 12 or 13.  The next page says:
> > 
> > Hits 26 - 35 out of 25
> > 
> > He should see 10, but only gets 5.
> > 
> > There's no way around this.  htsearch knows what hits will come up in the
> > next pages, but it has no way of passing that info to the mod_perl script
> > so that the mod_perl script can re-write the numbers.  The result is that
> > the user thinks things are broken because the number of hits reported are
> > significantly higher than what actually is returned. There are other
> > issues too, but they all revolve around this issue.
> > 
> > I've emailed Geoff (the lead developer of htdig) to see when the regexp
> > enabled restrict parameter will be available.  I should hear back soon.
>