Re: What happened?

[Roger Dingledine <arma@xxxxxxx>]

On Wed, Jan 26, 2005 at 01:02:41AM +0100, Kai Raven wrote:
> My personal tor indicator is my feed reader liferea with all feeds over privoxy
> & tor ;o) If all is OK with tor i get all my feeds. Since a few days i get a lot
> of HTTP Error Code 404: Ressource not found messages ;)

I think I've tracked one major source of these down: some of the larger
servers have been hitting the 1024 file descriptor limit, and things do
not fail gracefully when that happens.

This is good news, in that the Tor network really is handling tens of
thousands of streams simultaneously. Go us.

But it points to a bigger problem, too. As people continue to use Tor
for long-term connections like IRC and AIM, our big fast servers are
going to get clogged with streams that take up a file descriptor and a
few bytes of memory but not really much else. This is great, except for
some reason file descriptors seem to be considered rare commodities.

The short-term solution is to fix the bugs so when the connection is
refused, a "resource limit" reason is sent back, rather than the "misc
error, close the conn now" which is sent currently. That way the client
can try somewhere else. But if most servers are full most of the time,
performance will still be bad.

We're already in the process of integrating libevent, so we can make use
of epoll, kqueue, and whatever the other snazzy interfaces are that can
poll many file descriptors efficiently.

Can we just ask server operators to pump up their ulimit -n to a couple
hundred thousand? Or will this make people upset.

We could also implement per-port quotas, stop advertising ourselves in
the directory when a certain threshold percent of our file descriptors
are occupied, etc, if this stays a problem.

How do ulimits work on win32, anyway?

Any other suggestions?

--Roger