Re: [Libevent-users] Design struggle

On Thu, Sep 4, 2014 at 9:40 AM, Azat Khuzhin <a3at.mail@xxxxxxxxx> wrote:

On Fri, Aug 29, 2014 at 05:53:50PM +0200, B.R. wrote:
> 1Â) Filters are created cointaining information of connections they should
> be bound to (protocol, port), events that should trigger them and a

I guess only one of this: protocol || port

âThat was just a description, not a question actually.â

âFilters work on a specific protocol+port,â I require them to provide both pieces of information.

This allows me to:

- Automatically determine which port is to be listened to for which protocol (auto-determination of sockets to create/bind/listen to)

- Have multiple filters on the same tuple (protocol, port), sharing the read information and writing to the socket if an answer they can provide is needed

> I) If multiple filters ask to be triggered on READ event on the same
> socket, what will they see? Will they all be able to read the same message
> which triggered the event and not another one (ie another message received
> while processing the filter list)?

It depends from how you use this.

Kernel will just return array of bytes when you call read() on some
fd/socket, and once you read it, the second read() will never return the
same portion of data.

You could use evbuffer, to save data, and then feed it to every filter
that assigned to this fd/socket, and drain it when you finished.

âI cannot use evbuffer, though it was the initial (and the best) idea. Remember I support bot TCP and UDP.

Do evbuffers now handle UDP? The docs and user reports still indicate the contrary.â

>
> II) If multiple filters wanna write data to the socket, is it safe to have
> each filter having its separate buffer and triggering its own WRITE event
> on it?

ââ

This is not safe, because write(2) not always will write all you portion
of data, it could write less then @count, and return this value to
libevent, and then event loop will schedule the second WRITE event for
the same fd/socket:

write(10, foobar[4096]foobaz[4096], 8192) = 4096 # event1
write(10, barfoo[4096]bazfoo[4096], 8192) = 4096 # event2
write(10, foobaz[4096], 4096) = 4096 # event1
write(10, bazfoo[4096], 4096) = 4096 # event2
...
read(10, buf, 8192*2) = 8192*2

Actual:
buf = foobar[4096]barfoo[4096]foobaz[4096]bazfoo[4096]
Expected:
buf = foobar[4096]foobaz[4096]barfoo[4096]bazfoo[4096]

âActually, I moved on from than and, even if I am using separate buffers for incoming data, I share the write event (bound to the fd) among all the filters.

When writing to the buffer, I take caution not to overwrite existing data and use the buffer content counter to append data if some is already existing.

But yeah I have a problem: I cannot know which filter wrote which part of the buffer...

And I cannot multiplex writings to the output buffer since, as you pointed out, there is no way of knowing how the asynchronous system will dispatch the WRITE events to the socket.

Actually, it looks to me as a false problem:

- In UDP, there is no 'stream' concept, only single isolated datagrams. That is simple: for each packet, you decide to write something or not depending on its processing. There is no 'chunk' concept, and you write everything in one shot: there is no multiplexing, hence no other filter might interleave data.

- In TCP, you deal with a stream, and there is no way of knowing if sent data will be received in any number of packets. That is why you check bounds and if not all the data has been received, you then wait for more to come. The mistake here would be to partially process data and if possible send a partial answer: interleaving might happen from other filters, making the sent message gibberish.

Partial handling is a nice feature of stream sockets, however the asynchronous nature of their processing in our case means there is no control over the filters multiplexing. The other solution would be to attempt to totally order handlers to avoid interleaving, but that woudl require a global state, locking... and rendering this magnificent asynchronous system synchronous again... great. ^^

Since data is now appended to the same buffer, associated in a single event to the same socket fd, I suppose everything is totally ordered now. Am I right?

Do you notice any flaw there?

>
> Here is a use case summing up both previous inquiries:
> Say a filter wrote data to a socket after parsing the content it read on
> it, and that the peer reacted to that answer, what will subsequent filters,
> for which the READ event has been triggered by the initial message, read?
> a) Initial message?
> b) Either initial message or answer (race, undecided)?
> c) Nothing since the event has been canceled (not pending anymore), the
> subsequent filters will only receive a new event for READ on reception of
> the answer
>
> I am thinking of using a dispatcher which would sequentially (and manually)
> trigger the event of each of the filters. However that implies not linking

Why you want to do it manually?

âBecause binding several filters to the same fd means that only one of themâ will receive information.

Multiple recv/recvfrom on the same socket returns the received information for only one call, the others being empty.

I then recv/recvfrom once and populate individual filters buffer with it (by appending data). Now each filter can handle it safely and play with their own buffer content counter to indicate where new data shall be inserted by the data-receiving event handler.

> the filters event with the socket, thus forcing me to maintain a separate
> buffer for each of the filter (with memory and processing overhead that
> implies). Moreover, the problem reappears if another message is received on
> the socket while the dispatching loop is at work... and the sequential work
> makes the benefit of the event-based system disappear!

Which problem?
You mean that you filter-handling stuff works significantly slower than
read/write?

â
The problem of not being able to read the same information from the socket several times (which is normal).

Another part of the problem is what happens if one of the filters might be at work while several messages are received on the wire. I do not want any filter to miss any part of any received message.â

â I thought that, by handling the push of the received information to the filters buffer individually, after having checked that their own event (the one associated with their callback procedure)â

ââ is not active (running), I would prevent that loss.â

Here is a sample case:

received "message1"

triggers READ event on handler1 -> handler1 reads "message1" and processes it

triggers READ event on handler2 -> handler2 reads "message2" and processes it

received "message2"

triggers READ event on handler1 (still busy processing)

triggers READ event on handler2 -> handler2 reads "message2" and processes it

received "message3"

triggers READ event on handler1 -> handler1 reads "message3" and processes it

triggers READ event on handler2 -> handler2 reads "message3" and processes it

Whoops! handler1 missed "message2"...

>
> I do not know if the bufferevents would be useful to my case. Anyway, those
> are not an option, since part of the traffic I am handling is UDP datagrams.

âI said I was using UDP too. evbuffers seem to be of no help to me...

---
B. R.