Re: 1 major problem

[Jason Pincin <jpincin@ashtech.net>]

I'm very interested in what responses you get about the usability of 3.2.x

If he's not going to back port it, what is the release schedule looking like?




On Mon, Jun 14, 1999 at 09:57:31AM -0700, Aaron Turner wrote:
> 
> Recap:
> 
> Jason and I talked on Sunday via IRC about the search engine and what
> htdig can and can not do.  (I'll post the log of the IRC soon, which will
> make things more understandable below.) One of the big issues was that
> Jason wanted to be able to restrict the results of a search by the URL.  
> While htdig can do this, it can currently only match on one single string.
> 
> This means that you can't say:
> 
> restrict1=path1&restrict2=path2
> 
> or
> 
> restrict=/c=1\.3\.\d/
> 
> Basically what Jason wanted to do was in the case of:
> 
> c=1.2.5.9
> c=1.2.5.6
> c=1.2.5.20
> 
> to be able to tell htdig to only search two of those paths without the
> 3rd.  I thought I could hack some perl to remove matches that we didn't
> want.  Like if we wanted 1.2.5.9 and 1.2.5.20 but not 1.2.5.6, we'd
> search for 1.2.5 and remove all matches containing 1.2.5.6 post-htdig.
> 
> The problem is that for searches that return a lot of results, those
> results are likely to span multiple pages of results.  What happens is
> that htdig will return X results per page.  So if htdig returns 25
> results, and 1/2 of the hits are in 1.2.5.6 the page will say:
> 
> Search results 1 - 25 of 72:
> 
> But only have 13 results in the page.  This obviously is confusing
> to the user, and they're likely to think the search is broke.
> 
> I read on the htdig list archives, that they're working on a regexp
> enabled restrict paramter.  Ie you could say:
> 
> restrict=/(1\.2\.5\.9|1\.2\.5\.20)/
> 
> Looks like the regexp feature is complete in the 3.2 development tree
> (we're at 3.1.2 which is the latest stable).  Geoff isn't sure if he's
> going to back port it right now.  I'm going to ask around about what
> experiance people are having with 3.2 and if it's production ready or not.
> (My guess is no.)
> 
> Also, I realized that this problem is bigger than I first thought.  It not
> only happens when you use the Application option (as Jason and I discussed
> using it), but any time you do a search from high in the tree (such as
> c=1) and htdig finds duplicates because the same article lives in two
> different places in the tree. (Our goal is to remove the duplicates.)
> 
> There is only one way to really fix this problem:
> 
> Make sure that duplicate articles have the same URL.  This means we either
> strip the c= param from the URL or set c= to be the same for all cases.
> Ie:
> 
> /articles/article.php3?id=10&c=1.3.5.9 becomes:
> /articles/article.php3?id=10
> 
> I'm sure everyone sees the problem with this- you can't use the URL to
> drill down searches in htdig anymore.
> 
> Basically, we're screwed either way, and I don't know how to work around
> it.  I'm going to email the htdig list and see if anyone has a solution.
> 
> -Aaron
> 
> 
> On Mon, 14 Jun 1999, Jason Pincin wrote:
> 
> > Bummer... but what can ya do?  :)  Let me know what kind of response
> > you get from the htdig guys.
> > 
> > On Sun, Jun 13, 1999 at 09:28:43PM -0700, Aaron Turner wrote:
> > > 
> > > Ok, so there's one "major" problem with this whole post-htsesarch restrict
> > > idea.  Basically it will happen when many results are returned- more
> > > results than would fit on one web page.
> > > 
> > > Basically since we will be removing hits after htsearch has generated
> > > them, htsearch's understanding of how many hits there are will be
> > > incorrect.  Example:
> > > 
> > > Joe drills down to 1.3, selects Apache, and types in "performance" as his
> > > keyword.  htsearch finds 35 hits, the first 25 of which will appear on
> > > the first page, the remaining 10 on the second page.
> > > 
> > > Problem is that my mod_perl script will remove hits after htsearch figures
> > > out what hit belongs on what page.  So while the first page says:
> > > 
> > > Hits 1 - 25 out of 35
> > > 
> > > Maybe 1/2 of them are duplicates.  This means the user expects 25 hits on
> > > the page, but only sees 12 or 13.  The next page says:
> > > 
> > > Hits 26 - 35 out of 25
> > > 
> > > He should see 10, but only gets 5.
> > > 
> > > There's no way around this.  htsearch knows what hits will come up in the
> > > next pages, but it has no way of passing that info to the mod_perl script
> > > so that the mod_perl script can re-write the numbers.  The result is that
> > > the user thinks things are broken because the number of hits reported are
> > > significantly higher than what actually is returned. There are other
> > > issues too, but they all revolve around this issue.
> > > 
> > > I've emailed Geoff (the lead developer of htdig) to see when the regexp
> > > enabled restrict parameter will be available.  I should hear back soon.
> >