Re: Debugging

[Aaron Turner <aturner@linuxkb.org>]

On Mon, 6 Mar 2000, Jason Pincin wrote:

> > Reading this email, has me uh, concerned.  My reading, of this is that
> > this failed because there were slightly over 200 entries in the DB of type
> > "user".  And anytime we reach this magic number, we're going to have this
> > timeout problem.  One thing, why can't we increase the timeout?  There's
> > gotta be a setting.  
> 
> No.  Not 200 of type user.  What caused the timeout was the fact that
> there were 200 objects with the same parent.  When that parent is loaded,
> all of it's children get initialized with it.  Yes, we can increase the
> timeout... but it's pretty high currently, and I don't know if we want
> people waiting over a minute for a page anyway.

No, I got the "200 children of any kind".  That's what has me so
concerned.  What is the current time out?
 
> > Why do you think this is a PHP timeout issue?  What is the limit in terms
> > of time?  Would getting a faster DB system help?  A faster web server? I
> 
> Perhaps.  Who knows.  I know it's a PHP timeout issue... whether it's
> timing out due to lack of processing power, or lack of DB power... I'm
> not quite sure...

Well, let me put it this way.  We have faster CPU's than what my company
uses for MUCH larger datasets.  I'm talking MILLIONS of records searched
per-query.  Our query time is < 2 seconds.  Yes, we use Oracle, but I'd
have a hard time believing that MySQL can't beat Oracle when you're
comparing < 500 records to > 6 million.

Something is very wrong with this picture.  We're not scaling.  We're not
even close.  We have to find out what/why.

> > don't think re-writing this in C/C++ is something that will ever happen.
> > C/C++ is just too low level for this sort of thing.  And from the sounds
> 
> Yes it will be.  In case you've never listened to my rants about it on
> IRC, this is one of the ideas that interests me most.  To rewrite the
> object code in C/C++ as a stand-alone server with a modurlized back-end.
> Dan and I have discussed as well.  That way other projects can more
> easily utilize the system we've been designing... it's the whole ODS
> thing I talked with you about some time ago... it will happen, it's just
> a matter of when...

Ok, let me re-phrase that.  Rewriting this in C/C++ won't happen in time
to help us.  Even if you stoped all PHP devel, and started on the C++
port, you're at least 18, probably closer to 36, months away to having
anything useful.  Not to mention everything that we have coded would be
thrown away.

I'm going to the engineering office of my company this week.  We've got
some really smart people who deal with scaling issues day in and day out.  
I'll talk to a good friend of mine there, and see if he can give me some
pointers of how to find out what might be going on.  He's really good at
determining bottlenecks and suggesting solutions.

> > of it, I have this bad feeling that Zend won't be ready in time for us
> > either.  Which means examining other options; sooner rather than later.
> 
> I'm not sure Zend won't make it in time.  The problem surfaces when you
> get lots of objects under one category... and right now, content isn't
> even being added... so we have a little bit of time.  I'll also be
> looking at how I can optimize things in the current code... it's quite
> possible I haven't written the most optimized block of code possible
> *shrug*.

Four things:

1) Don't bet the farm on vapor-ware.  I know Zend is hardly vaporware, but
we don't have the ability to test it right now, and no idea when critical
bugs will be fixed.

2) Our goal of growing the content is directly in conflict with the
problem.  We need more content to get more visitors.  This needs to be a
priority for us.

3) You don't sound very confident that Zend will scale to meet our needs
in the forseeable future.

4) Other options?  My gut says, Perl as it performs well (especially with
DB's) and is high-level enough where it won't take 3 years to develop a
usable site.  mod_perl is known to scale very well; slashdot, internet
movie database, Wired, and CBS MarketWatch all run mod_perl.  You might
want to check out Mason.  http://www.masonhq.com/  From what I understand,
kinda a Zope meets mod_perl.

--
Aaron Turner, Core Developer       http://vodka.linuxkb.org/~aturner/
Linux Knowledge Base Organization  http://www.linuxkb.org/
Because world domination requires quality open documentation.
aka: aturner@vicinity.com, aturner@pobox.com, ion_beam_head@ashtech.net
The difference between `Unstable' and `Usable' is only two characters: NT