Re: [Libevent-users] http client throttle

To: libevent-users@xxxxxxxxxxxxx

Subject: Re: [Libevent-users] http client throttle

From: Jesper Lundgren <koudelkaa@xxxxxxxxx>

Date: Mon, 20 Jul 2015 09:56:41 +0800

Delivered-to: archiver@xxxxxxxx

Delivered-to: libevent-users-outgoing@xxxxxxxx

Delivered-to: libevent-users@xxxxxxxx

Delivery-date: Sun, 19 Jul 2015 21:57:04 -0400

Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:content-type; bh=1sOBYx/xbr7qrhbrsR+Xm8Iyp3biQCaZdxC2yBqIngU=; b=01UlccYPk44l9Pn9zGx2LN5R2CTFRWa2wxEU/Hn/BO6CQssnB2NBEy1EJVFwyPz6Hk GX+nMCUBC3SAZwZPl7//9RjoZhFvO27JRRcTzKfTAFt2TIdPtDZ/N3viLOyS2T3oGhLY IW/EMniPpkEQ8BVt2jKFvIQMmCPhuZyategfKxAiQgM+VwvX5/S42AVsFEuBHfv2O/tz 8SyBwO7QvxAm5MyFRH9CiQrZgpvrKwzEse8bsI1aH5rGJMs/vyjxbZrmrtgFRgkXS5B5 gKVCJGGXTyJIZmVT3AvvTHfVjIwlqd5HgUkNtg3F9PN8fADQ2PM5srPnHYHWRuvJMGK7 a8SQ==

In-reply-to: <CAHdZv9CKRxs_GXBzcbJH=qz8q=b6hvMgpzHh_Q37grpRK1tY2g@xxxxxxxxxxxxxx>

References: <CADevSdoA57xxO2gNXj3Xu0ri4zFwhNkn3+eYhfXiK+FQKjswfg@xxxxxxxxxxxxxx> <CAHdZv9CKRxs_GXBzcbJH=qz8q=b6hvMgpzHh_Q37grpRK1tY2g@xxxxxxxxxxxxxx>

Reply-to: libevent-users@xxxxxxxxxxxxx

Sender: owner-libevent-users@xxxxxxxxxxxxx

Hi Mark, and thank you for your suggestion. Are you aware of any example code that uses this technique to dynamically throttle requests? I've looked at the bufferevent rate limit options but it looks more bandwidth oriented and static in nature.Â

On a second thought. Why is libevent http returning NULL requests with 36 Operation now in progress errors to me? I have set connection to unlimited retry so I thought it would deal with these issues (and throttle appropriately in case the server can not handle and returns errors).

Am I really supposed to have to deal with these issues?

On Mon, Jul 20, 2015 at 9:06 AM, Mark Ellzey <mthomas@xxxxxxxxxx> wrote:

You can either set a timer to ease up the requests/s, or use the rate-limiting functionality.Â
Use "struct buffer event * evhttp_connection_get_bufferevent", and read the section on
on this api here:Â
http://www.wangafu.net/~nickm/libevent-book/Ref6a_advanced_bufferevents.html

On Sun, Jul 19, 2015 at 3:30 AM Jesper Lundgren <koudelkaa@xxxxxxxxx> wrote:
Hello, I am using libevent http in client mode to send GET requests. Many of the requests I try to send end with http_req_done callbacks where req == NULL and socket error is set to 36 (Operation now in progress). My guess is that the server can not handle the traffic but I am not sure what is the best approach to dynamically adjust the request rate in libevent to prevent socket errors.

Any suggestions?

/JL