Hi Martin
I wasn't sure, if I should report that as a bug. The new version of [tcpserver] seems to work fine, BUT now it suffers again from the initial problem, that maxlib's [netserver] is still suffering from:
If the send buffer of a certain socket is full, [tcpserver] is blocked and thus blocks Pd.
From a user perspective, i can think of a few solutions to deal with
that. However, i don't have clue, what's possible/difficult to implement.
a) If the server doesn't get the necessary ACKs in a meaningful time, if shuts down the connection after a while.
b) Instead of writing to the send buffer, [tcpserver] could output a message saying, that the data couldn't be sent.
c) Change the behaviour of the 'sent' message from last outlet so, that it reflects the amount of bytes actually transmitted and not the number of bytes written to the send buffer.
d) <put something here, that i didn't think of>
cheers Roman
Roman Haefeli wrote:
Hi Martin
Hi Roman, sorry for the late reply.
I wasn't sure, if I should report that as a bug. The new version of [tcpserver] seems to work fine, BUT now it suffers again from the initial problem, that maxlib's [netserver] is still suffering from:
If the send buffer of a certain socket is full, [tcpserver] is blocked and thus blocks Pd.
From a user perspective, i can think of a few solutions to deal with
that. However, i don't have clue, what's possible/difficult to implement.
a) If the server doesn't get the necessary ACKs in a meaningful time, if shuts down the connection after a while.
It's up to the OS to deal with ACKs. The programmer has no (easy) way of accessing the low-level TCP communications. The send() function puts the data in the queue and returns. If the queue is blocked you don't find out until much later. The connection is closed some minutes after the disconnect occurs.
b) Instead of writing to the send buffer, [tcpserver] could output a message saying, that the data couldn't be sent.
Except it doesn't know the data couldn't be sent. I tried unplugging the cable and then plugging it back in. Sometimes all the data that was sent in the meantime will suddenly appear at the client, a few minutes after it was sent from the server. Other times I get 'client disconnected' messages a few minutes after the cable was unplugged. The timeout could probably be adjusted with the Time To Live parameter for the socket but it won't ever be zero. The internet protocols were designed to work over broken networks so it's normal that they wouldn't fail at the first attempt to send. UDP was intended for 'send and forget' messages. Maybe you could try [udpsend] to send vast quantities of data, alongside [tcpserver] for control. The previous incarnation of [tcpserver] used select() to see if there is still room in the buffer but as we saw this slows the whole thing to a crawl since the buffer has to be checked for every single byte in case it just filled up. It seems more efficient to have the client reply when it gets the message.
c) Change the behaviour of the 'sent' message from last outlet so, that it reflects the amount of bytes actually transmitted and not the number of bytes written to the send buffer.
Again 'it thinks' the data _was_ sent, so it returns the amount that it sent, even if it is still in the queue on the server, as long as the connection is still open.
d) <put something here, that i didn't think of>
I think you need to incorporate some kind of feedback from the clients. Either each client could send a request or a heartbeat every second or so, or the server could require a response to each message before sending more. Or use a separate UDP channel for the kind of data that could rapidly fill a buffer, like audio.
Martin
Hi Martin
On Sat, 2009-12-12 at 14:09 -0500, Martin Peach wrote:
Hi Roman, sorry for the late reply.
Complex matters require time.
I wasn't sure, if I should report that as a bug. The new version of [tcpserver] seems to work fine, BUT now it suffers again from the initial problem, that maxlib's [netserver] is still suffering from:
If the send buffer of a certain socket is full, [tcpserver] is blocked and thus blocks Pd.
From a user perspective, i can think of a few solutions to deal with
that. However, i don't have clue, what's possible/difficult to implement.
a) If the server doesn't get the necessary ACKs in a meaningful time, if shuts down the connection after a while.
It's up to the OS to deal with ACKs. The programmer has no (easy) way of accessing the low-level TCP communications. The send() function puts the data in the queue and returns. If the queue is blocked you don't find out until much later. The connection is closed some minutes after the disconnect occurs.
I see.
b) Instead of writing to the send buffer, [tcpserver] could output a message saying, that the data couldn't be sent.
Except it doesn't know the data couldn't be sent. I tried unplugging the cable and then plugging it back in. Sometimes all the data that was sent in the meantime will suddenly appear at the client, a few minutes after it was sent from the server. Other times I get 'client disconnected' messages a few minutes after the cable was unplugged. The timeout could probably be adjusted with the Time To Live parameter for the socket but it won't ever be zero. The internet protocols were designed to work over broken networks so it's normal that they wouldn't fail at the first attempt to send. UDP was intended for 'send and forget' messages. Maybe you could try [udpsend] to send vast quantities of data, alongside [tcpserver] for control. The previous incarnation of [tcpserver] used select() to see if there is still room in the buffer but as we saw this slows the whole thing to a crawl since the buffer has to be checked for every single byte in case it just filled up. It seems more efficient to have the client reply when it gets the message.
c) Change the behaviour of the 'sent' message from last outlet so, that it reflects the amount of bytes actually transmitted and not the number of bytes written to the send buffer.
Again 'it thinks' the data _was_ sent, so it returns the amount that it sent, even if it is still in the queue on the server, as long as the connection is still open.
d) <put something here, that i didn't think of>
I think you need to incorporate some kind of feedback from the clients. Either each client could send a request or a heartbeat every second or so, or the server could require a response to each message before sending more. Or use a separate UDP channel for the kind of data that could rapidly fill a buffer, like audio.
As _I_ understand TCP (i might be wrong), it is supposed to handle all transport-only related issues, such as correct order, completeness etc. Of course, i could adapt the protocol i use to what i think is supposed to be part of the transport layer, but because i think, that checking the responsiveness (or availability, actually) of the clients is part of the transport and not the application layer, i am not going to introduce that change.
I see that it is very complex matter and personally i understand too little of it in order to have an idea how those issues could be addressed (in [tcpserver], for instance).
My personal experience is, that i tried to write a netpd-server in plain python based on the module SocketServer. I finally managed to do it, but i ended up with a server, that suffered from the exact same problem: When wfile.write method didn't return because of a full socket buffer, the whole server was blocked. People on #python advised me not to use SocketServer directly, which most people find hard to deal with because of such issues, but to use the python-twisted framework instead. It was dead-easy to do it in twisted; basically all i had to do was to implement the netpd-protocol. Twisted does handle all nasty issues quite gracefully. If on a specific socket no ACKs are received for a certain while (or until the send buffer is full, i don't know exactly), it just disconnects the client. If the client would be available after that again, it would just receive the content of the buffer from the server and then disconnect. Both ends notice, that something went wrong, and that they would have re-establish the connection from scratch in order to get to the similar state as before. In terms of netpd, this means, that all patches' states are not confirmed, so the client has to request a state dump again. This all can be handled consistently, since the client can be sure, that it either receives everything or if not, it _certainly_ gets disconnected.
If i would know how threading works (in python) and how to use, i think would try to implement it this way: Each socket would have its own queue. Whenever the server wants to send a message to one or more clients, it would put the message on the socket's queue. Each socket connection would run in its own thread and tries to send the message from the queue to its respective client. If the wfile.write() method wouldn't return immediately (let's say after some milliseconds time-out), it would try it again after some time. If the queue of a certain client reaches a certain limit size, the respective client would get disconnected. I haven't really tried that approach (haven't digged that deep into python) and probably it is not as easy as it may sound, but when I heard of the new [tcpserver] I thought it might be something along those lines. I really don't mean to sound smart-assy, but do you think, such an approach could be feasible for [tcpserver]?
Cheerio Roman