(Windows 10, Pd-50.0)
i'm sending messages of 25 bytes with [mrpeach/tcpserver] to a client (NodeMCU/ESP8266),
triggered by [metro 50].
with a tcpclient in a seperate Pd the messages are received in the right order, no gaps.
on the Node i get sometimes 1, most of the times multiple messages in one read.
the sending is done with "broadcast", because in the future there will be more clients.
how come the [mrpeach/tcpclient] and the WifiClient on the Node give different results,
and can tcpserver be 'forced' to treat them equally?
any hint is welcome.
rolf
On Sun, 2020-06-14 at 22:26 +0200, IOhannes m zmölnig wrote:
On 2020-06-14 22:22, rolfm@dds.nl wrote:
and can tcpserver be 'forced' to treat them equally?
any hint is welcome.
not really helping, but does the same happen with iemnet/tcpserver?
Surely not, since iemnet tpc classes output a stream of floats and do not try to "guess" some kind of message.
Roman
On 2020-06-15 17:05, Roman Haefeli wrote:
On Sun, 2020-06-14 at 22:26 +0200, IOhannes m zmölnig wrote:
On 2020-06-14 22:22, rolfm@dds.nl wrote:
and can tcpserver be 'forced' to treat them equally?
any hint is welcome.
not really helping, but does the same happen with iemnet/tcpserver?
Surely not, since iemnet tpc classes output a stream of floats and do not try to "guess" some kind of message.
ah, obviously i was confused about the actual problem (so much for the "domain experts").
but then, not really: it's true that the iemnet's tcp-objects will output single bytes (rather than packet-like lists) when *receiving* data, but the question here seems to be about *sending*.
of course, in reality the problem is most likely about mrpeach/tcpclient vs the node-client (rather than mrpeach/tcpserver, as the subject suggests). and here the problem is (most likely) not that one client outputs the data in different chunks than the other, but that you try to send *packets* over a a TCP/IP connection without further delimiting.
this just cannot work.
it can't work due to the nature of TCP/IP (which is a stream-protocol, and doesn't know anything about packets). of course, sometimes it *does* work, like any buggy software that is worth its buck. but in the general case it won't.
which is the reason why iemnet's tcp-objects output this impractical single-byte messages: to force the user to deal with the problem from the beginning (e.g. by using mrpeach's fabulous [slipenc]/[slipdec] objects) rather than wait for the house to fall apart one the day before the show.
gasmdr IOhannes
On Sun, Jun 14, 2020 at 4:24 PM rolfm@dds.nl wrote:
(Windows 10, Pd-50.0)
i'm sending messages of 25 bytes with [mrpeach/tcpserver] to a client (NodeMCU/ESP8266),
triggered by [metro 50].
with a tcpclient in a seperate Pd the messages are received in the right order, no gaps.
Are both Pds on the same machine? (I'm guessing yes)
on the Node i get sometimes 1, most of the times multiple messages in one read.
Are they multiples of the same message or a sequence? (I'm guessing a sequence)
the sending is done with "broadcast", because in the future there will be more clients.
Does it also happen if you send only to that client? (I'm guessing yes)
how come the [mrpeach/tcpclient] and the WifiClient on the Node give different results,
and can tcpserver be 'forced' to treat them equally?
Based on my assumptions, it may be that the OS is delaying sending the messages in case you're not finished sending them. You need a break of some minimum time before the whole lot gets sent. The OS may delay longer in the case of a wireless connection to avoid network congestion. This is because TCP has no concept of packet or 'datagram' like UDP -- you can keep sending until you close the connection, so one way is to close the connection after each message. Another way would be to use a longer metro tick period so the stack times out and sends. Also try sending a longer message to force it to be sent.
Martin
any hint is welcome.
rolf
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Based on my assumptions, it may be that the OS is delaying sending the messages in case you're not finished sending them.
I think that's why Pd (and iemnet) sets the TCP_NODELAY socket option on TCP sockets .
Christof
On 15.06.2020 15:44, Martin Peach wrote:
On Sun, Jun 14, 2020 at 4:24 PM rolfm@dds.nl wrote:
(Windows 10, Pd-50.0)
i'm sending messages of 25 bytes with [mrpeach/tcpserver] to a client (NodeMCU/ESP8266),
triggered by [metro 50].
with a tcpclient in a seperate Pd the messages are received in the right order, no gaps.
Are both Pds on the same machine? (I'm guessing yes)
on the Node i get sometimes 1, most of the times multiple messages in one read.
Are they multiples of the same message or a sequence? (I'm guessing a sequence)
the sending is done with "broadcast", because in the future there will be more clients.
Does it also happen if you send only to that client? (I'm guessing yes)
how come the [mrpeach/tcpclient] and the WifiClient on the Node give different results,
and can tcpserver be 'forced' to treat them equally?
Based on my assumptions, it may be that the OS is delaying sending the messages in case you're not finished sending them. You need a break of some minimum time before the whole lot gets sent. The OS may delay longer in the case of a wireless connection to avoid network congestion. This is because TCP has no concept of packet or 'datagram' like UDP -- you can keep sending until you close the connection, so one way is to close the connection after each message. Another way would be to use a longer metro tick period so the stack times out and sends. Also try sending a longer message to force it to be sent.
Martin
any hint is welcome.
rolf
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On Mon, Jun 15, 2020 at 10:11 AM Christof Ressi info@christofressi.com wrote:
Based on my assumptions, it may be that the OS is delaying sending the messages in case you're not finished sending them.
I think that's why Pd (and iemnet) sets the TCP_NODELAY socket option on TCP sockets .
Right! I have that on tcpreceive and tcpsend but didn't add it to the tcpclient and tcpserver for some reason. I will fix that.
Martin
Christof
On 15.06.2020 15:44, Martin Peach wrote:
On Sun, Jun 14, 2020 at 4:24 PM rolfm@dds.nl wrote:
(Windows 10, Pd-50.0)
i'm sending messages of 25 bytes with [mrpeach/tcpserver] to a client (NodeMCU/ESP8266),
triggered by [metro 50].
with a tcpclient in a seperate Pd the messages are received in the right order, no gaps.
Are both Pds on the same machine? (I'm guessing yes)
on the Node i get sometimes 1, most of the times multiple messages in one read.
Are they multiples of the same message or a sequence? (I'm guessing a sequence)
the sending is done with "broadcast", because in the future there will be more clients.
Does it also happen if you send only to that client? (I'm guessing yes)
how come the [mrpeach/tcpclient] and the WifiClient on the Node give different results,
and can tcpserver be 'forced' to treat them equally?
Based on my assumptions, it may be that the OS is delaying sending the messages in case you're not finished sending them. You need a break of some minimum time before the whole lot gets sent. The OS may delay longer in the case of a wireless connection to avoid network congestion. This is because TCP has no concept of packet or 'datagram' like UDP -- you can keep sending until you close the connection, so one way is to close the connection after each message. Another way would be to use a longer metro tick period so the stack times out and sends. Also try sending a longer message to force it to be sent.
Martin
any hint is welcome.
rolf
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Hi Rold, Hi Martin
On Mon, 2020-06-15 at 09:44 -0400, Martin Peach wrote:
Based on my assumptions, it may be that the OS is delaying sending the messages in case you're not finished sending them. You need a break of some minimum time before the whole lot gets sent.
That probably forces the underlying framework to create messages of the desired format. What if you want to send many messages immediately?
The OS may delay longer in the case of a wireless connection to avoid network congestion. This is because TCP has no concept of packet or 'datagram' like UDP -- you can keep sending until you close the connection, so one way is to close the connection after each message. Another way would be to use a longer metro tick period so the stack times out and sends.
How can you know what is a time that works (over wifi, vpn, ethernet)? That is not using the TCP protocol as intended.
Also try sending a longer message to force it to be sent.
What a horrible advice! Just stuff more unnecessary data into to pipe to force it to put delimiters at the right spot!
Seriously, what you lack is some kind of delimiting mechanism in order to be able to transmit packets through a stream-oriented protocol such as TCP or serial line. mrpeach tcp classes do a great job in obfuscating the true nature of TCP by outputting randomly assembled lists of floats instead of a stream of float (like iemnet).
OTOH, mrpeach comes with [slipenc] and [slipdec] which implement SLIP which is probably the simplest available delimiting mechanism.
This should work:
[send <your message encoded as list of bytes>(
[slipenc]
[tcpclient]
[tcpserver]
[slipdec]
[print]
I'm sure there is some implementation of SLIP for the NodeMCU somewhere.
Things would get more complicated when you receive TCP data from many clients in parallel, but you seem to want to only send to many clients.
An alternative approach would be to use UDP instead of TCP. UDP thinks already in packets, but chances are that packets are lost on the way, while TCP is absolutely reliable.
Roman
TL;DR: [iemnet/tcpserver] + [mrpeach/slipenc] + [mrpeach/slipdec] :-)
Depending on your payload, you might also get away with FUDI (ASCII text delimited by semicolons), which is used internally by Pd and slightly easier to parse (but more limited).
I've said this elsewhere: as much as I like (and use) the mrpeach library, I avoid the net objects (especially [tcpclient] and [tcpserver]) because there are too many problems in the code. [iemnet], on the other hand, has been (re)written and maintained by actual domain experts.
Christof
On 15.06.2020 17:29, Roman Haefeli wrote:
Hi Rold, Hi Martin
On Mon, 2020-06-15 at 09:44 -0400, Martin Peach wrote:
Based on my assumptions, it may be that the OS is delaying sending the messages in case you're not finished sending them. You need a break of some minimum time before the whole lot gets sent.
That probably forces the underlying framework to create messages of the desired format. What if you want to send many messages immediately?
The OS may delay longer in the case of a wireless connection to avoid network congestion. This is because TCP has no concept of packet or 'datagram' like UDP -- you can keep sending until you close the connection, so one way is to close the connection after each message. Another way would be to use a longer metro tick period so the stack times out and sends.
How can you know what is a time that works (over wifi, vpn, ethernet)? That is not using the TCP protocol as intended.
Also try sending a longer message to force it to be sent.
What a horrible advice! Just stuff more unnecessary data into to pipe to force it to put delimiters at the right spot!
Seriously, what you lack is some kind of delimiting mechanism in order to be able to transmit packets through a stream-oriented protocol such as TCP or serial line. mrpeach tcp classes do a great job in obfuscating the true nature of TCP by outputting randomly assembled lists of floats instead of a stream of float (like iemnet).
OTOH, mrpeach comes with [slipenc] and [slipdec] which implement SLIP which is probably the simplest available delimiting mechanism.
This should work:
[send <your message encoded as list of bytes>( [slipenc] [tcpclient]
[tcpserver] [slipdec] [print]
I'm sure there is some implementation of SLIP for the NodeMCU somewhere.
Things would get more complicated when you receive TCP data from many clients in parallel, but you seem to want to only send to many clients.
An alternative approach would be to use UDP instead of TCP. UDP thinks already in packets, but chances are that packets are lost on the way, while TCP is absolutely reliable.
Roman
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On Mon, Jun 15, 2020 at 12:32 PM Christof Ressi info@christofressi.com wrote:
I've said this elsewhere: as much as I like (and use) the mrpeach library, I avoid the net objects (especially [tcpclient] and [tcpserver]) because there are too many problems in the code. [iemnet], on the other hand, has been (re)written and maintained by actual domain experts.
I have revised the code in those objects and just uploaded the whole mrpeach for Windows10 to deken. (this includes fixes to tcpserver and tcpclient as well as midifile) So anyway I have lots of free time now and I am maintaining the library, in the sourceforge svn repo (https://sourceforge.net/p/pure-data/svn/HEAD/tree/trunk/externals/mrpeach/). If there were a git repo for externals I would use that but it seems scattered at the moment.
Martin
On 2020-06-15 19:46, Martin Peach wrote:
If there were a git repo for externals I would use that but it seems scattered at the moment.
that's actually a feature not a bug :-) the idea is, that smaller, independent and self-contained repositories are more agile and resilient than a huge mono-repository.
also, since we have deken people can have their code scattered however they like and still the users can actually find them.
in any case, if you are looking for a place to put your externals and if you are not afraid of using gitlab, a good starting point could be https://github.com/pd-externals/ if you want to i can either add you to the team or create one or more repositories ("mrpeach"?) and add you only to those. and of course, you could also just use your own personal namespace.
gsamrd IOhannes
I have revised the code in those objects
I see some bug fixes and the REUSEADDR and NODELAY options, that's cool!
If you have lots of free time, you also might want to revise the whole threading code. Spawning a new thread for every list of bytes is not only excessively wasteful (especially in a soft-realtime environment), but also breaks determinism: if you send list A and list B in short succession, thread A is not guaranteed to call "send()" before thread B, so bytes might arrive out of order on the other side. Instead push the bytes to a ringbuffer (preferrably lockfree) and let a *single* sender thread consume it.
Christof
On 15.06.2020 19:46, Martin Peach wrote:
On Mon, Jun 15, 2020 at 12:32 PM Christof Ressi info@christofressi.com wrote:
I've said this elsewhere: as much as I like (and use) the mrpeach library, I avoid the net objects (especially [tcpclient] and [tcpserver]) because there are too many problems in the code. [iemnet], on the other hand, has been (re)written and maintained by actual domain experts.
I have revised the code in those objects and just uploaded the whole mrpeach for Windows10 to deken. (this includes fixes to tcpserver and tcpclient as well as midifile) So anyway I have lots of free time now and I am maintaining the library, in the sourceforge svn repo (https://sourceforge.net/p/pure-data/svn/HEAD/tree/trunk/externals/mrpeach/). If there were a git repo for externals I would use that but it seems scattered at the moment.
Martin
On Mon, Jun 15, 2020 at 3:09 PM Christof Ressi info@christofressi.com wrote:
I have revised the code in those objects
I see some bug fixes and the REUSEADDR and NODELAY options, that's cool!
If you have lots of free time, you also might want to revise the whole threading code. Spawning a new thread for every list of bytes is not only excessively wasteful (especially in a soft-realtime environment), but also breaks determinism: if you send list A and list B in short succession, thread A is not guaranteed to call "send()" before thread B, so bytes might arrive out of order on the other side. Instead push the bytes to a ringbuffer (preferrably lockfree) and let a *single* sender thread consume it.
Yes, that's next. I first wanted to see if I have stopped the crashes that seemed to be due to non-threadsafe calls. The one crash report I've seen shows it occurring inside Pd's main event loop because of a call to clockdelay from another thread while Pd is manipulating the same clock. If it works, It's nearly ready to move to a single thread, I already use a thread to receive and check for completed sends. Should be ready by next week :) .
Martin
Christof
On 15.06.2020 19:46, Martin Peach wrote:
On Mon, Jun 15, 2020 at 12:32 PM Christof Ressi info@christofressi.com wrote:
I've said this elsewhere: as much as I like (and use) the mrpeach library, I avoid the net objects (especially [tcpclient] and [tcpserver]) because there are too many problems in the code. [iemnet], on the other hand, has been (re)written and maintained by actual domain experts.
I have revised the code in those objects and just uploaded the whole mrpeach for Windows10 to deken. (this includes fixes to tcpserver and tcpclient as well as midifile) So anyway I have lots of free time now and I am maintaining the library, in the sourceforge svn repo (https://sourceforge.net/p/pure-data/svn/HEAD/tree/trunk/externals/mrpeach/). If there were a git repo for externals I would use that but it seems scattered at the moment.
Martin
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Sounds good!
On 15.06.2020 21:41, Martin Peach wrote:
On Mon, Jun 15, 2020 at 3:09 PM Christof Ressi info@christofressi.com wrote:
I have revised the code in those objects
I see some bug fixes and the REUSEADDR and NODELAY options, that's cool!
If you have lots of free time, you also might want to revise the whole threading code. Spawning a new thread for every list of bytes is not only excessively wasteful (especially in a soft-realtime environment), but also breaks determinism: if you send list A and list B in short succession, thread A is not guaranteed to call "send()" before thread B, so bytes might arrive out of order on the other side. Instead push the bytes to a ringbuffer (preferrably lockfree) and let a *single* sender thread consume it.
Yes, that's next. I first wanted to see if I have stopped the crashes that seemed to be due to non-threadsafe calls. The one crash report I've seen shows it occurring inside Pd's main event loop because of a call to clockdelay from another thread while Pd is manipulating the same clock. If it works, It's nearly ready to move to a single thread, I already use a thread to receive and check for completed sends. Should be ready by next week :) .
Martin
Christof
On 15.06.2020 19:46, Martin Peach wrote:
On Mon, Jun 15, 2020 at 12:32 PM Christof Ressi info@christofressi.com wrote:
I've said this elsewhere: as much as I like (and use) the mrpeach library, I avoid the net objects (especially [tcpclient] and [tcpserver]) because there are too many problems in the code. [iemnet], on the other hand, has been (re)written and maintained by actual domain experts.
I have revised the code in those objects and just uploaded the whole mrpeach for Windows10 to deken. (this includes fixes to tcpserver and tcpclient as well as midifile) So anyway I have lots of free time now and I am maintaining the library, in the sourceforge svn repo (https://sourceforge.net/p/pure-data/svn/HEAD/tree/trunk/externals/mrpeach/). If there were a git repo for externals I would use that but it seems scattered at the moment.
Martin
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
-
Christof Ressi -
IOhannes m zmölnig -
Martin Peach -
rolfm＠dds.nl -
Roman Haefeli