On Sep 3, 2012, at 8:13 PM, Parvez Shaikh wrote: Purpose of doing this is to make only one thread access socket descriptor.
I have to use multiple threads(thread pool that does some mundane processing of messages) for utilizing multiple cores as much as possible(each thread access no shared resource except queuing message for send); also I don't want multiple threads doing "send" operation on same socket descriptor - which may require lot of lock-unlock operations(around send) and in absence of lock-unlock data interleaving would create trouble (I am using STREAM sockets).
Sound reasonable.
What I would like to do is to have a single thread - an event reactor thread(that dispatches events) access my socket descriptor and does read and write.
I would do the same. For read I am using "bufferevents" for their ease of use. However for write I can't use "bufferevents" (I don't want my send operations to be delayed,
I don't think your send operations will be delayed by the buffer event. I would start by using buffer event for both receiving and sending messages.
further even with multiple messages in bufferevent leads to only one invocation of read callback - as per my observations - which is not suitable for me - I'd like for every message that is buffered to have read callback called.)
The read callback should loop over the messages in the bufferevent and invoke your application "message received" event with each message.
There are two approaches for this -
1. write callback on socket descriptor would notify me when descriptor is "write-ready", when I will pump data until I fail with EAGAIN(this would mean send buffer full). I will return from write callback and would pump data when descriptor is "write-ready" again.
2. I would define a separate (custom) event on socket(no EV_READ or EV_WRITE just EV_PERSIST with no timeout). A thread that enqueues data for "send" activates the event after enqueuing. An event callback would access the data that was enqueued and pump it on descriptor until EAGAIN. I will return from event's callback and would try pumping data again when data is enqueued/
These are not separate approaches - you must use both if you want to pump out messages as soon as possible.
First, the socket is ready for writing, and there are no messages in the queue - write event should not be enabled in this case. The event loop is waiting for read event.
Then some messages enter the message queue - you must wake up the event loop to drain your message queue. Depending on the amount of data and the connection to the other machine, you may get EAGAIN before the message queue is empty.
If no new message enter the queue, and you don't enable EV_WRITE event, the rest of messages in the queue may sit there for unlimited time. So you want to enable EV_WRITE, and the event loop will wake and continue to drain your messages.
Somehow I am inclined to option 1 (tried option 2 and it worked) because libevent tells write readiness (when probability of EAGAIN would be less at least for first write) and in option 2 there is danger that every time I try to send data I may get EAGAIN for many messages leading to wasted event activation (rare but possible?).
If you get too many write failures, you can write only when not waiting for write readiness. You will have to collect stats when running with real load to see if this is a problem.
However, if you get lot of write failures, it means that you send too much data and the real problem is how to limit and monitor your message queue size.
On Mon, Sep 3, 2012 at 7:03 PM, Nir Soffer <nirsof@xxxxxxxxx> wrote: Thanks Oleg and Nir.
I am also pursuing another approach. Here I have defined a custom event (associated with FD but not having either EV_READ or EV_WRITE) and there is no time out either.
A thread queues data to "outgoing message list" and activates an event. In response to this, event reactor thread (one with dispatch) calls a function that sends message from this list one by one until "send" encounters EAGAIN.
Requesting your feedback on this approach.
If you must create your messages in another thread (maybe it is cpu bound, or you must use blocking operations), this should work. But if you don't have too, I would not use threads since I don't like deadlocks :-)
Anyway, when the send fails with EAGAIN, you want to enable EV_WRITE, so you can continue to drain your messages queue when socket is ready. The way you work with EV_WRITE is not related to working with threads. On Sun, Sep 2, 2012 at 3:45 PM, Nir Soffer <nirsof@xxxxxxxxx> wrote: On Sun, Sep 2, 2012 at 6:54 AM, Parvez Shaikh <pshaikh.world@xxxxxxxxx> wrote: Thanks Oleg,
Second approach is what I am doing.
Why disabled EV_WRITE in write callback?
I'd wish to have this callback called again whenever send buffer has space, so disabling EV_WRITE will prevent this.
You want to enable EV_WRITE only when you have something to write.
When you enable EV_WRITE, libevent will add the descriptor to the the event backend (e.g select). When the descriptor is ready for writing, the write callback will be called. If you always enable EV_WRITE, the callback will be called on every loop iteration, and the event loop will never wait for events, since the descriptor is almost always ready for writing. On Fri, Aug 31, 2012 at 3:54 PM, Oleg <mybrokenbeat@xxxxxxxxx> wrote: EV_WRITE calls whenever you can write to send buffer until it's not full.
So if you have never called send(), but EV_WRITE is enabled you will receive this event each new loop (your CPU will be ~100% used).
If you have called send() and it didn't return EAGAIN you will also receive this event next loop.
If you have called send() and it returned EAGAIN you will receive this event when your send buffer will have some free space.
So for your sending queue it should look like this in pseudo code:
1) Application->Send(data)
{
if (send() != EAGAIN )
return;
else
{
queue.push(data);
enable(EV_WRITE,onWriteCallback);
}
}
2) onWriteCallback()
{
while(!queue.empty)
{
if (send(queue.front())== EAGAIN);
return;
queue.pop();
}
disable(EV_WRITE);
}
31.08.2012, Ð 8:50, Parvez Shaikh ÐÐÐÐÑÐÐ(Ð):
> Hi all,
>
> I have a question about EV_WRITE event as to when does it's callback function invoked?
>
> Is it that when someone first executes write on an fd associated with EV_WRITE event?
>
> Or when libevent detects that application can now write to fd without getting errors?
>
> For EV_READ it is easy to understand that it's callback is invoked when data is available for read on fd but not clear about EV_WRITE.
>
> Here is what I am trying to do -
>
> I am trying to write asynchronous send/recv application; in which I will read data on connected sockets asynchronously using "EV_READ's callback.
>
> For send however, I will enqueue the data to be sent in my own queue(application will write data to this buffer) and I will flush the buffer in EV_WRITE callback.
>
> Now if I get the error EAGAIN in send operation in callback of EV_WRITE, I will simply return and on next invocation of EV_WRITE's callback I will start flushing my buffer again.
>
> Thanks,
> Parvez
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users in the body.
|