Re: [Libevent-users] EV_WRITE - Wait for a socket or FD to become writeable.

[Parvez Shaikh <pshaikh.world@xxxxxxxxx>]

Hi Nir,

Purpose of doing this is to make only one thread access socket descriptor.

I have to use multiple threads(thread pool that does some mundane processing of messages) for utilizing multiple cores as much as possible(each thread access no shared resource except queuing message for send); also I don't want multiple threads doing "send" operation on same socket descriptor - which may require lot of lock-unlock operations(around send) and in absence of lock-unlock data interleaving would create trouble (I am using STREAM sockets).

What I would like to do is to have a single thread - an event reactor thread(that dispatches events) access my socket descriptor and does read and write. For read I am using "bufferevents" for their ease of use. However for write I can't use "bufferevents" (I don't want my send operations to be delayed, further even with multiple messages in bufferevent leads to only one invocation of read callback - as per my observations - which is not suitable for me - I'd like for every message that is buffered to have read callback called.)

There are two approaches for this -
1. write callback on socket descriptor would notify me when descriptor is "write-ready", when I will pump data until I fail with EAGAIN(this would mean send buffer full). I will return from write callback and would pump data when descriptor is "write-ready" again.

2. I would define a separate (custom) event on socket(no EV_READ or EV_WRITE just EV_PERSIST with no timeout). A thread that enqueues data for "send" activates the event after enqueuing. An event callback would access the data that was enqueued and pump it on descriptor until EAGAIN. I will return from event's callback and would try pumping data again when data is enqueued/

Somehow I am inclined to option 1 (tried option 2 and it worked) because libevent tells write readiness (when probability of EAGAIN would be less at least for first write) and in option 2 there is danger that every time I try to send data I may get EAGAIN for many messages leading to wasted event activation (rare but possible?).

Thanks,
Parvez

On Mon, Sep 3, 2012 at 7:03 PM, Nir Soffer <nirsof@xxxxxxxxx> wrote:

On Mon, Sep 3, 2012 at 1:05 PM, Parvez Shaikh <pshaikh.world@xxxxxxxxx> wrote:

Thanks Oleg and Nir.

I am also pursuing another approach. Here I have defined a custom event (associated with FD but not having either EV_READ or EV_WRITE) and there is no time out either.

A thread queues data to "outgoing message list" and activates an event. In response to this, event reactor thread (one with dispatch) calls a function that sends message from this list one by one until "send" encounters EAGAIN.

Requesting your feedback on this approach.

If you must create your messages in another thread (maybe it is cpu bound, or you must use blocking operations), this should work. But if you don't have too, I would not use threads since I don't like deadlocks :-)

Anyway, when the send fails with EAGAIN, you want to enable EV_WRITE, so you can continue to drain your messages queue when socket is ready. The way you work with EV_WRITE is not related to working with threads.

Â

On Sun, Sep 2, 2012 at 3:45 PM, Nir Soffer <nirsof@xxxxxxxxx> wrote:

On Sun, Sep 2, 2012 at 6:54 AM, Parvez Shaikh <pshaikh.world@xxxxxxxxx> wrote:

Thanks Oleg,

Second approach is what I am doing.

Why disabled EV_WRITE in write callback?

I'd wish to have this callback called again whenever send buffer has space, so disabling EV_WRITE will prevent this.

You want to enable EV_WRITE only when you have something to write.

When you enable EV_WRITE, libevent will add the descriptor to the the event backend (e.g select). When the descriptor is ready for writing, the write callback will be called. If you always enable EV_WRITE, the callback will be called on every loop iteration, and the event loop will never wait for events, since the descriptor is almost always ready for writing.

Â

On Fri, Aug 31, 2012 at 3:54 PM, Oleg <mybrokenbeat@xxxxxxxxx> wrote:

EV_WRITE calls whenever you can write to send buffer until it's not full.
So if you have never called send(), but EV_WRITE is enabled you will receive this event each new loop (your CPU will be ~100% used).
If you have called send() and it didn't return EAGAIN you will also receive this event next loop.
If you have called send() and it returned EAGAIN you will receive this event when your send buffer will have some free space.


So for your sending queue it should look like this in pseudo code:

1) Application->Send(data)
{
    if (send() != EAGAIN )
        return;
    else
    {
        queue.push(data);
        enable(EV_WRITE,onWriteCallback);
    }
}

2) onWriteCallback()
{
    while(!queue.empty)
    {
        if (send(queue.front())== EAGAIN);
            return;
        queue.pop();
    }
    disable(EV_WRITE);
}

31.08.2012, Ð 8:50, Parvez Shaikh ÐÐÐÐÑÐÐ(Ð):

> Hi all,
>
> I have a question about EV_WRITE event as to when does it's callback function invoked?
>
> Is it that when someone first executes write on an fd associated with EV_WRITE event?
>
> Or when libevent detects that application can now write to fd without getting errors?
>
> For EV_READ it is easy to understand that it's callback is invoked when data is available for read on fd but not clear about EV_WRITE.
>
> Here is what I am trying to do -
>
> I am trying to write asynchronous send/recv application; in which I will read data on connected sockets asynchronously using "EV_READ's callback.
>
> For send however, I will enqueue the data to be sent in my own queue(application will write data to this buffer) and I will flush the buffer in EV_WRITE callback.
>
> Now if I get the error EAGAIN in send operation in callback of EV_WRITE, I will simply return Âand on next invocation of EV_WRITE's callback I will start flushing my buffer again.
>
> Thanks,
> Parvez

***********************************************************************
To unsubscribe, send an e-mail to majordomo@xxxxxxxxxxxxx with
unsubscribe libevent-users  Âin the body.