I've been trying to debug this really annoying issue. The [info complete] look in pd_readsocket was causing some drastic slowdowns on some GUI objects like Scope~, making them unusable. If you remove the [info complete] bracket from pd_readsocket, there are these intermittent Tcl stacktraces causes by messages from 'pd' to 'pd-gui' that get split in the wrong spot.
I have found one very interesting result: the break always happens at 48k. At least when I test with the bitmap-madness.pd patch that comes with tclpd. It is a good test patch because it causes heavy pd --> pd-gui traffic. So I've tried a bunch of things and not much really seems to affect it:
- in configure_socket, change fconfigure -buffering to "line" - in configure_socket, add -buffersize 500000 to fconfigure - in s_inter.c set GUI_ALLOCCHUNK anywhere from 1k to 64k
It seemed that lowering GUI_ALLOCCHUNK did make it happen more often, but always the break was exactly at 48k.
Anyone have any ideas?
.hc
Quick guess; the socket itself (at the OS level) has a fixed buffer size and if the sending process exceeds it it is suspended until the receiving process eats at least some of it. The receiving process is thus handed a truncated buffer.
There's not much way around this. One possibility (if indeed this is a serious efficiency issue) would be for Pd to append a "done" message to each batch up tcl to up-send. On the receiving end you could then just 'string search' to the last occurrence of the special string and send that text safely to the interpreter.
cheers M
On Sun, Jan 22, 2012 at 11:42:01PM -0500, Hans-Christoph Steiner wrote:
I've been trying to debug this really annoying issue. The [info complete] look in pd_readsocket was causing some drastic slowdowns on some GUI objects like Scope~, making them unusable. If you remove the [info complete] bracket from pd_readsocket, there are these intermittent Tcl stacktraces causes by messages from 'pd' to 'pd-gui' that get split in the wrong spot.
I have found one very interesting result: the break always happens at 48k. At least when I test with the bitmap-madness.pd patch that comes with tclpd. It is a good test patch because it causes heavy pd --> pd-gui traffic. So I've tried a bunch of things and not much really seems to affect it:
- in configure_socket, change fconfigure -buffering to "line"
- in configure_socket, add -buffersize 500000 to fconfigure
- in s_inter.c set GUI_ALLOCCHUNK anywhere from 1k to 64k
It seemed that lowering GUI_ALLOCCHUNK did make it happen more often, but always the break was exactly at 48k.
Anyone have any ideas?
.hc
Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
Le 2012-01-22 à 21:16:00, Miller Puckette a écrit :
Quick guess; the socket itself (at the OS level) has a fixed buffer size and if the sending process exceeds it it is suspended until the receiving process eats at least some of it.
AFAIK, that's 4k or 8k for «unix» type (pipe or fs socket) and it might be 32k for TCP sockets. But code must not rely on specific buffer sizes.
If you remove the [info complete] bracket from pd_readsocket, there are these intermittent Tcl stacktraces causes by messages from 'pd' to 'pd-gui' that get split in the wrong spot.
yes, [info complete] is a potential major slowdown because it may reparse code many times while the amount of code increases. It's a n² time order thing.
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
On Jan 23, 2012, at 12:41 AM, Mathieu Bouchard wrote:
Le 2012-01-22 à 21:16:00, Miller Puckette a écrit :
Quick guess; the socket itself (at the OS level) has a fixed buffer size and if the sending process exceeds it it is suspended until the receiving process eats at least some of it.
AFAIK, that's 4k or 8k for «unix» type (pipe or fs socket) and it might be 32k for TCP sockets. But code must not rely on specific buffer sizes.
If you remove the [info complete] bracket from pd_readsocket, there are these intermittent Tcl stacktraces causes by messages from 'pd' to 'pd-gui' that get split in the wrong spot.
yes, [info complete] is a potential major slowdown because it may reparse code many times while the amount of code increases. It's a n² time order thing.
The [info complete] loop had the advantage of actually fixing this problem. The problem wasn't so much just using [info complete] but the condition used to trigger the check. What was happening was that Scope~ was sending some kind of blank lines a lot, and the "while {![info complete]}" looped fast over these blank lines trying to make Tcl code out of them, eating up a lot of CPU. I figured I might as well try to solve the underlying problem rather than tacking on workaround on top of workaround.
.hc
----------------------------------------------------------------------------
I hate it when they say, "He gave his life for his country." Nobody gives their life for anything. We steal the lives of these kids. -Admiral Gene LeRocque
On Mon, Jan 23, 2012 at 6:41 AM, Mathieu Bouchard matju@artengine.ca wrote:
If you remove the [info complete] bracket from pd_readsocket, there are these intermittent Tcl stacktraces causes by messages from 'pd' to 'pd-gui' that get split in the wrong spot.
yes, [info complete] is a potential major slowdown because it may reparse code many times while the amount of code increases. It's a n² time order thing.
moreover, one should not trust [info complete] as an oracle knowing exactly where a Tcl command ends and another begins. it just checks for unbalanced brackets/quotes [1].
think about: .x8763 configure -text blah \n -command {blah…} [info complete ".x8763 configure -text blah"] would return 1, resulting in the evaluation of "-command {blah…}" as a separate command".
Le 2012-01-23 à 18:57:00, mescalinum@gmail.com a écrit :
moreover, one should not trust [info complete] as an oracle knowing exactly where a Tcl command ends and another begins. it just checks for unbalanced brackets/quotes [1].
Right. But if only reading complete lines, then a trailing backslash is a sign of an incomplete command. Apparently, [info complete \] returns 1.
think about: .x8763 configure -text blah \n -command {blah…} [info complete ".x8763 configure -text blah"] would return 1, resulting in the evaluation of "-command {blah…}" as a separate command".
But a \n wouldn't happen there, it would be a \\n, I mean a single backslash immediately followed by a newline, or else the whole thing would be enclosed in more braces, brackets or quotes.
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
Le 2012-01-22 à 21:16:00, Miller Puckette a écrit :
There's not much way around this. One possibility (if indeed this is a serious efficiency issue) would be for Pd to append a "done" message to each batch up tcl to up-send.
That's called a newline... not preceded by a backslash. The thing with fconfigure -buffering line is that it doesn't care about backslashes, whereas eval does, and if you use both together, you need to account for that difference.
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
As it stands right now, there are new-lines in the middle of certain tcl commands sent from Pd, so if newline were it, I'd have to go chase down all the pretty-printing newlines in the Pd code and delete them. That might actually might work; I'd only be worried taht some extern was throwing up tcl code with intra-tcl-statement newlines that would occasionally throw false positives at the tcl completeness checker, if indeed all it did was check that the tcl buffer ended in a newline.
cheers Miller
On Mon, Jan 23, 2012 at 12:45:09AM -0500, Mathieu Bouchard wrote:
Le 2012-01-22 à 21:16:00, Miller Puckette a écrit :
There's not much way around this. One possibility (if indeed this is a serious efficiency issue) would be for Pd to append a "done" message to each batch up tcl to up-send.
That's called a newline... not preceded by a backslash. The thing with fconfigure -buffering line is that it doesn't care about backslashes, whereas eval does, and if you use both together, you need to account for that difference.
| Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
Le 2012-01-22 à 21:54:00, Miller Puckette a écrit :
On Mon, Jan 23, 2012 at 12:45:09AM -0500, Mathieu Bouchard wrote:
That's called a newline... not preceded by a backslash. The thing with fconfigure -buffering line is that it doesn't care about backslashes, whereas eval does, and if you use both together, you need to account for that difference.
Sorry, the problem is indeed more complex than just looking for backslashes before newlines. It has to do with counting braces. The most efficient way to do it would be a parser that keeps its own state so that after returning from [info complete] you don't need to call it again from scratch, but that does not exist in Tcl (it's possible that there's a Tcl extension that you could compile for that, but I wouldn't know about it).
The way Pd is designed, sys_gui has never any obligation to contain a whole statement, so there isn't much that you could do in sys_gui to help the situation for all externals at once. If it were the case, you could for example use nul-terminators ('\0' aka (char)0) to make the parsing much easier on the receiving end.
However, if you make the assumption that any statement started within a wb-function must be terminated in the same wb-function, then the caller of the wb-function can call some kind of cleanup that adds a nul-terminator iff number of bytes written by sys_gui >= 0 (since last termination). Same goes with t_guicallbackfn.
But there are a number of cases of sys_gui being called from elsewhere. Considering end of t_clock run and end of t_pollfn run as being ends of statement could help, as well as _setup function. Is there any other case that wouldn't be covered by that ?
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
On Jan 23, 2012, at 12:42 PM, Mathieu Bouchard wrote:
Le 2012-01-22 à 21:54:00, Miller Puckette a écrit :
On Mon, Jan 23, 2012 at 12:45:09AM -0500, Mathieu Bouchard wrote:
That's called a newline... not preceded by a backslash. The thing with fconfigure -buffering line is that it doesn't care about backslashes, whereas eval does, and if you use both together, you need to account for that difference.
Sorry, the problem is indeed more complex than just looking for backslashes before newlines. It has to do with counting braces. The most efficient way to do it would be a parser that keeps its own state so that after returning from [info complete] you don't need to call it again from scratch, but that does not exist in Tcl (it's possible that there's a Tcl extension that you could compile for that, but I wouldn't know about it).
The way Pd is designed, sys_gui has never any obligation to contain a whole statement, so there isn't much that you could do in sys_gui to help the situation for all externals at once. If it were the case, you could for example use nul-terminators ('\0' aka (char)0) to make the parsing much easier on the receiving end.
However, if you make the assumption that any statement started within a wb-function must be terminated in the same wb-function, then the caller of the wb-function can call some kind of cleanup that adds a nul-terminator iff number of bytes written by sys_gui >= 0 (since last termination). Same goes with t_guicallbackfn.
But there are a number of cases of sys_gui being called from elsewhere. Considering end of t_clock run and end of t_pollfn run as being ends of statement could help, as well as _setup function. Is there any other case that wouldn't be covered by that ?
The problem isn't related to sys_gui not enforcing full Tcl commands since the breaks don't happen along the lines of sys_gui command contents. The only time I've seen a problem is when the commands are broken in the middle of a sys_gui post.
Here's a transcript of my discussion on ##tcl which has some things to try:
Hans-Christoph Steiner I'm having a weird socket buffering issue in my app, Pd. It is two processes: a pure Tcl GUI (pd-gui) and a C engine (pd) which communication over a TCP socket. In pd-gui, I setup the socket to get the data like this: fconfigure $sock -blocking 0 -buffering line -encoding utf-8; fileevent $sock readable {::pd_connect::pd_readsocket ""}
then pd_readsocket reads from the socket using [read $pd_socket]. pd sends lots of Tcl commands to pd-gui which are then executed in pd_readsocket by doing:
if {[catch {uplevel #0 $cmd_from_pd} errorname]} {
Mostly this works well, but occasionally, the $cmd_from_pd contents are not broken at the new line, but instead split in this middle of a Tcl command.
<thommey> you could use [gets] instead of [reads] and process by-line <Gotisch> shouldnt -buffering line prevent that? <Gotisch> i mean the problem he is descripting
Hans-Christoph Steiner actually, mostly, we use "-buffering none"
<Gotisch> descripting! <Gotisch> terrible
Hans-Christoph Steiner but "-buffering line" doesn't seem to change the behavior
<Gotisch> well shouldn't -buffering line do exactly that <Gotisch> mh maybe it just means <Gotisch> be only readable if at least a line is ready to be read
aku Hans - The OS is free to break packets wherever it wants. You cannot rely on the data in one puts arriving as a single read. This is outsaide of Tcl's control. You have to [read] into a buffer, and use [info complete] to see when a command is complete and can be executed.
Hans-Christoph Steiner the funny little detail is that when it breaks in the middle of a Tcl command, it is always exactly at the 48kb mark. exactly.
<Gotisch> i dont understand it then <Gotisch> "If newValue is line, then the I/O system will automatically flush output for the channel whenever a newline character is output."
aku Some OS setting I guess, or default, or whatnot. The -buffering on the sender side only controls when Tcl pushes data into the OS. It doesn't prevent the OS from packaging things as it likes
<Gotisch> doesnt that mean you get the readable directly after end of line? <Gotisch> oh <Gotisch> well then i would go with thommeys suggestion
aku [gets] can help, assuming that each command is a single line. If not, you still have to accumulate lines until [info complete] on the buffers singlas that the command is complete
<Gotisch> would need to check for linebreaks in buffer too though <Gotisch> or you could go from 0.5 commands to 1.5 commands and so on <Gotisch> well very unrealistic scenario
Hans-Christoph Steiner unfortunately, there can be commands with escaped newlines in them, so I don't think that [gets] will work
<Gotisch> i dont think gets would stop at escaped new lines
Hans-Christoph Steiner oh, really? something to try
<thommey> [gets] performs no output processing, you will still have to check for \ at the end of the line yourself and then read the next one and buffer
Hans-Christoph Steiner thommey: that seems doable, better than running [info complete] each time, I suppose <Gotisch> i guess i didnt know what i was talking about again
Hans-Christoph Steiner the 48k size doesn't ring any bells for anyone?
<Gotisch> maybe its the buffer size <Gotisch> on the c side that is
Hans-Christoph Steiner I was playing around with the buffring on the C/pd side, and that didn't change the 48k break size
<Gotisch> ah
Hans-Christoph Steiner I changed it anywhere from 1k to 128k
kbk It's someone's buffer limit. Whose, not sure. But the solutions that aku is describing will work. Use [gets] and [info complete] -- and be prepared to handle the [fblocked] return from [gets]
Hans-Christoph Steiner kbk: so this [gets] technique, I would need to "fconfigure -blocking 1"? would I still use:
fileevent $sock readable {::pd_connect::pd_readsocket ""} ?
aku No, it is still -blocking 0 on the receiver side. Small example, see: http://www.activestate.com/blog/2010/05/concurrency-tcl-events-without-getti... [gets] returns -1 if the line is incomplete. You can then simply try again at the next event i.e. see if the line is has become complete.
.hc
----------------------------------------------------------------------------
If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of everyone, and the receiver cannot dispossess himself of it. - Thomas Jefferson
Le 2012-01-23 à 13:31:00, Hans-Christoph Steiner a écrit :
The problem isn't related to sys_gui not enforcing full Tcl commands since the breaks don't happen along the lines of sys_gui command contents.
I know that sys_gui doesn't enforce it, but that's not what I'm trying to say. I'm saying that it would be good if sys_gui could send extra data such as \0 terminators or size-of-upcoming command so that you don't need to run [info complete] at all in the tcl process.
A TCP connection doesn't have actual breaks. This means that if you want breaks, you have to add them. Currently, the breaks have to be guessed by a combination of parsing and supposing that the OS's breaks will eventually match an end-of-statement before the RAM starts swapping or before the CPU goes quadratic. That's why I'm thinking about less implicit ways of ending statements.
The only time I've seen a problem is when the commands are broken in the middle of a sys_gui post.
And there are several ways to fix that, and I was speculating about one that we aren't usually thinking about, but that might save the effort of running [info complete] multiple times on the same data and any other kludges we have to combine it with so that [info complete] does the job, and other funny side-effects of an [info complete]-based strategy...
the funny little detail is that when it breaks in the middle of a Tcl command, it is always exactly at the 48kb mark. exactly.
This most probably has to do with the size of the buffers in the TCP lasagna of the Linux kernel or OSX kernel. Each OS has its own sizes, and in some of them, it's tunable, if you know how (either by playing with /proc or /sys, or by recompiling something with a modified config.h, if you're not lucky).
A direct pipe (the pipe() command, no port number ; or a /tmp socket-file) has usually a different buffer size, iirc. In any case, you can't really rely on the buffer size being anything in particular, but the way it is now, it may affect Pd, especially in certain tough situations.
Some OS setting I guess, or default, or whatnot. The -buffering on the sender side only controls when Tcl pushes data into the OS. It doesn't prevent the OS from packaging things as it likes
Yeah, [fconfigure -buffering] is FOR OUTPUT ONLY. (If I said anything else, it's a mistake). However, [fconfigure -buffersize] controls the size of both the input and output buffers.
An input buffer is always required for using [gets], but is not when using [read]. This is because it's extremely inefficient to look for a newline when the OS doesn't allow you to «unread» data. I'm talking about user-space buffers (the OS does the buffering it wants to have, but you don't have direct access to it).
Hans-Christoph Steiner thommey: that seems doable, better than running [info complete] each time, I suppose <Gotisch> i guess i didnt know what i was talking about again
The thing with the \ is that it's not sufficient, because you also have to count the braces, and braces can be backslashed. (This corresponds to a mistake that I made in a previous mail.)
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
On Jan 23, 2012, at 12:54 AM, Miller Puckette wrote:
On Mon, Jan 23, 2012 at 12:45:09AM -0500, Mathieu Bouchard wrote:
Le 2012-01-22 à 21:16:00, Miller Puckette a écrit :
There's not much way around this. One possibility (if indeed this is a serious efficiency issue) would be for Pd to append a "done" message to each batch up tcl to up-send.
That's called a newline... not preceded by a backslash. The thing with fconfigure -buffering line is that it doesn't care about backslashes, whereas eval does, and if you use both together, you need to account for that difference.
As it stands right now, there are new-lines in the middle of certain tcl commands sent from Pd, so if newline were it, I'd have to go chase down all the pretty-printing newlines in the Pd code and delete them. That might actually might work; I'd only be worried taht some extern was throwing up tcl code with intra-tcl-statement newlines that would occasionally throw false positives at the tcl completeness checker, if indeed all it did was check that the tcl buffer ended in a newline.
cheers Miller
Ok, so I tried an implementation based on [gets] that is line-by-line, so it gets a line, then executes it. This version is ridiculously slow, like barely usable. I'm now trying a version of this that uses [gets] to read everything in the buffer, then executes it in blocks, it seems faster, let's see if it fixes the problem!
Here's the line-by-line execution version:
proc ::pd_connect::pd_readsocket {cmd_from_pd} { variable pd_socket if {[eof $pd_socket]} { # if we lose the socket connection, that means pd quit, so we quit close $pd_socket exit } if {[gets $pd_socket line] < 0} {return} append cmd_from_pd $line\n if {[string index $cmd_from_pd end] eq "\"} { ::pd_connect::pd_readsocket $cmd_from_pd } if {[catch {uplevel #0 $cmd_from_pd} errorname]} { global errorInfo switch -regexp -- $errorname { "missing close-brace" { pd_readsocket $cmd_from_pd } "^invalid command name" { ::pdwindow::fatal [concat [_ "(Tcl) INVALID COMMAND NAME: "] $errorInfo "\n"] } default { ::pdwindow::fatal [concat [_ "(Tcl) UNHANDLED ERROR: "] $errorInfo "\n"] } } } }
.hc
----------------------------------------------------------------------------
If nature has made any one thing less susceptible than all others of exclusive property, it is the action of the thinking power called an idea, which an individual may exclusively possess as long as he keeps it to himself; but the moment it is divulged, it forces itself into the possession of everyone, and the receiver cannot dispossess himself of it. - Thomas Jefferson
Le 2012-01-23 à 14:00:00, Hans-Christoph Steiner a écrit :
if {[string index $cmd_from_pd end] eq "\"} { ::pd_connect::pd_readsocket $cmd_from_pd } if {[catch {uplevel #0 $cmd_from_pd} errorname]} {
This is very wrong, because pd_readsocket calls itself recursively N times, then doesn't return, which causes eval (uplevel) to be run N+1 times on the same chunk of statements.
This may cause «canvas create» to create very many duplicates of canvas items, and such.
Is that right ?
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
On Feb 7, 2012, at 3:37 PM, Mathieu Bouchard wrote:
Le 2012-01-23 à 14:00:00, Hans-Christoph Steiner a écrit :
if {[string index $cmd_from_pd end] eq "\"} { ::pd_connect::pd_readsocket $cmd_from_pd } if {[catch {uplevel #0 $cmd_from_pd} errorname]} {
This is very wrong, because pd_readsocket calls itself recursively N times, then doesn't return, which causes eval (uplevel) to be run N+1 times on the same chunk of statements.
This may cause «canvas create» to create very many duplicates of canvas items, and such.
Is that right ?
This code has totally changed, check master in pd-extended.git for the current code. The new code fixed this bug, and seems faster than the old code to boot.
.hc
----------------------------------------------------------------------------
"It is convenient to imagine a power beyond us because that means we don't have to examine our own lives.", from "The Idols of Environmentalism", by Curtis White
Le 2012-02-07 à 16:09:00, Hans-Christoph Steiner a écrit :
This code has totally changed,
I know, it was just in case you wanted to know. Your most recent mail didn't mention this bug (though it did say that you had removed the recursion).
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
Ok, I think I nailed it with lots of help from #tcl on freenode. It not only seems to fix the intermittent problem, but also seems to be a bit faster, due to using Tcl properly. I attached the patch I just committed to pd-extended. The while {[info complete]} loop did a lot of extra work because it was running again and again on the same chunk. That is mostly avoided by first checking if the block ends in a '\n'. In my tests, I never saw [info complete] get called or the "missing close-brace" error get thrown, everything was caught by the test for '\n' at the end. Then, I removed all [read] calls and recursive calls to [pd_readsocket] because they'll just be working on an empty buffer. Instead the code now waits for the next [fileevent], where Tcl tells us that there is indeed more data in the buffer.
Here's a lightly edited transcript of the session:
(02:32:35 PM) _hc: thommey kbk Gotisch aku: ok, I changed the implementation to use [gets] and the problem is the same (but in general execution seems a little faster than my previous approach using [read]) (02:33:01 PM) _hc: I still get the occasional breaks at the wrong place at the 48k mark (02:34:23 PM) _hc: http://pastebin.com/trwpnkg7 (02:34:26 PM) _hc: thats' the proc (02:34:39 PM) _hc: I tried -buffering none and line (02:34:45 PM) _hc: I tried -buffersize 1000000 14:35 (02:35:12 PM) Setok left the room (quit: Quit: Setok). (02:38:44 PM) ijchain: <aku> As I said, the OS may not deliver the whole thing to you in one packet / readable callback. After the loop 11-14 you do not know of the command is complete yet, or not, without checking via [info complete]. (02:39:22 PM) ijchain: <dgp> syntactical completeness and message completeness are different things as well. 14:40 (02:41:05 PM) ijchain: <dgp> perhaps [gets] has already taken care of that by delivering only compelte lines. (02:41:20 PM) ijchain: <aku> The recursive calls of the handler do not look good to me either, not really. Because we are at a point where [gets] returned -1, so we should not expect that we immediately get more data in the recursive call. (02:41:21 PM) _hc: I think [gets] is supposed to do that (02:41:49 PM) _hc: the recursive handler wasn't called at all in my tests (02:41:50 PM) ijchain: <dgp> Is the protocol defined so that only newline is used to terminate commands, and never the semicolon? (02:42:09 PM) _hc: the gets protocol? (02:42:24 PM) ijchain: <dgp> the protocol by which these two programs communicate over the socket. (02:43:17 PM) ijchain: <dgp> If unconstrained Tcl scripts are the unit of communication, be sure you handle semicolons correctly according to the needs of your program. (02:43:21 PM) _hc: the protocol from C->tcl is literally UTF-8 Tcl code (02:43:28 PM) _hc: that is executed (02:43:40 PM) _hc: see the uplevel (02:44:38 PM) ijchain: <aku> Is it possible for a single command to span multiple lines ? (02:44:42 PM) _hc: semi-colons are handled. Its an old program, started in '95. We just recently wrote the GUI side of it. Previuosly we used C to handle the networking on the Tcl side too, but I switched the code to pure Tcl (02:44:44 PM) ijchain: <dgp> yes (02:44:58 PM) _hc: this bug has only arisen in the pure Tcl implementation 14:45 (02:45:06 PM) ijchain: <dgp> set var {a (02:45:06 PM) ijchain: <dgp> b} (02:45:23 PM) ijchain: <dgp> or, more commonly... (02:45:33 PM) _hc: the problematic breaks always come in the middle of commands, let me post some examples, one sec (02:45:41 PM) ijchain: <dgp> proc foo {} { (02:45:41 PM) ijchain: <dgp> many (02:45:41 PM) ijchain: <dgp> lines (02:45:41 PM) ijchain: <dgp> here (02:45:41 PM) ijchain: <dgp> } (02:46:09 PM) ijchain: <dgp> "middle of commands" means not at a newline? (02:46:54 PM) Moryanta [Timmy@avalon.zpm.be] entered the room. (02:48:47 PM) ijchain: <dgp> program written in '95 == pre-unicode (02:48:54 PM) _hc: http://pastebin.com/1kuntv3m (02:49:04 PM) _hc: yup, we ported it to UTF-8 (02:49:32 PM) awb [~awb@12.206.216.226] entered the room. (02:49:37 PM) _hc: this problem is intermittent, it doesn't happen in the same place everytime, sometimes the same code will work fine 14:50 (02:50:06 PM) ijchain: <dgp> help me with the paste. (02:50:07 PM) _hc: if you look at the pastebin, that's exactly the code from C(pd) --> Tcl(pd-gui) (02:50:12 PM) ijchain: <dgp> what should I be looking at? (02:50:21 PM) _hc: if you scroll down to the long list of numners (02:50:26 PM) _hc: that's a long Tcl command (02:50:40 PM) _hc: the spot where there are some blank lines is where it gets cut sometimes (02:50:52 PM) ijchain: <aku> And where is the break ? (02:50:56 PM) ijchain: <aku> x-ed, ok (02:51:03 PM) _hc: so the Tcl stacktrace says "(Tcl) INVALID COMMAND NAME: invalid command name "450" (02:51:03 PM) _hc: while executing" (02:51:10 PM) ijchain: <dgp> what happened at line 119 ? (02:51:16 PM) _hc: yup, line 119 (02:51:31 PM) _hc: and that mark always lines up to be a 48k chunk of code (02:51:45 PM) ijchain: <aku> Well, the loop lines 11-14 in your code has obviously collected only the lines up to 118 and the next fileevent call got 119+ (02:51:46 PM) _hc: in the test that I have right now at least (02:51:47 PM) ijchain: <dgp> This is the data as sent? as received? (02:52:07 PM) _hc: aku, it works most of the time, and only sometimes gets cut there (02:52:18 PM) _hc: dgp that is the data received by Tcl (02:52:40 PM) ijchain: <dgp> our logic of splicing things together must be faulty (02:52:41 PM) ijchain: <aku> That is why I tell you three times now, that you have to check your buffer with [info complete], and wait for the next fileevent call to supply the rest of the command (02:52:47 PM) ijchain: <dgp> not taking escaped newlines into account (02:53:10 PM) _hc: aku: we had code that used [info complete] in it before, but it slowed things down a lot (02:53:26 PM) ijchain: <dgp> correctness, then speed (02:53:44 PM) _hc: a lot, like in hampering usability (02:53:52 PM) _hc: usability, then correctness, then speed (02:53:57 PM) ijchain: <dgp> no (02:54:00 PM) ijchain: <aku> What I see in your code is that your buffer (cmd_from_pd) is not global/persistent between fileevents also, which you need for that. (02:54:02 PM) qru: http://pastebin.com/1sfNZiLq (02:54:28 PM) ijchain: <dgp> Your solution will probably end up being stop inserting the escaped newlines. (02:54:31 PM) qru: That works to auth with a splunk server in http 2.5 but not 2.7. Any ideas? (02:54:43 PM) ijchain: <dgp> That's handy for readable code for humans. Does nothing here. (02:54:54 PM) _hc: at line 60, you can see our [info complete] implementaion: http://pure-data.git.sourceforge.net/git/gitweb.cgi?p=pure-data/pure-data;a=... 14:55 (02:55:00 PM) ijchain: <dgp> % info complete "foo bar\\n" (02:55:01 PM) ijchain: <dgp> 0 (02:55:06 PM) ijchain: <aku> Are you checking info complate after each line received, or after the full block has been adeed to the buffer ? (02:55:31 PM) _hc: blocks (02:55:36 PM) _hc: but that was with [read] (02:55:39 PM) ijchain: <dgp> qru, just a sec (02:57:22 PM) ijchain: <dgp> qru, what's the value of $::http::strict when working, and when not working? (02:57:59 PM) _hc: the thing about this issue is that its intermittent, most of the time everything works fine (02:58:16 PM) _hc: but when it fails, it seems to fail in a regular way (02:58:25 PM) _hc: i.e. the 48k block (02:58:32 PM) _hc: maybe that's misleading (02:58:52 PM) qru: dgp: I've tried both 1 and 0 with -strict and ::http::geturl (02:59:13 PM) ijchain: <aku> networking and the slicing and dicing of a character stream into packets, and re-assembly etc is non-deterministic. Sometimes things align, sometimes they don't. (02:59:31 PM) qru: Auth logs from Splunk show the attempt and the username is correctly parsed. (02:59:46 PM) qru: And the encoding is identical for the passwd. (02:59:52 PM) qru: In each case. (02:59:53 PM) ijchain: <aku> Looking at the read/info complete code. ... 15:00 (03:00:13 PM) ijchain: <aku> I do not understand why you are trying multiple reads in the pd_readsocket. (03:02:00 PM) ijchain: <aku> I.e. If the data is not complete you should from the handler. You have to save cmd_from_pd of course, so that you can add to it on the next call of the handler. (03:02:13 PM) ijchain: <aku> s/should from/should return from (03:03:40 PM) ***ijchain ferrieux has become available (03:04:33 PM) ijchain: <aku> With [gets] you can get away with reading multiple lines in the handler, until [gets] signals that the line is not complete. But with [read] without a limit as in your code you have the whole data the OS provided at this moment and a second read will not provide more. 15:05 (03:05:07 PM) ijchain: <dgp> The busy [read] loop might explan perceived poor performance though. (03:05:57 PM) ijchain: <dgp> aku, isn't the [eof] test before reading a misstep as well? (03:07:16 PM) ijchain: <aku> Not really, because if [read] generated eof you see it at the _next_ handler call. It makes code easier to write, with checks and exceptions first, then read and handling the result of that and able to simply return (03:07:51 PM) ijchain: <aku> in the processing, no need to bring everything back to a single point where we check eof (03:11:11 PM) _hc: ok, that bit about waiting for the next fileevent before calling [read] is interesting, I'll try that (03:11:27 PM) _hc: any thoughts on whether [read] or [gets] will be more efficient? (03:11:35 PM) ijchain: <dgp> get it right first (03:12:15 PM) ijchain: <dgp> _hc, I think the logic is wrong (03:12:31 PM) ijchain: <dgp> hmmm....not sure. (03:12:32 PM) _hc: dgp: which logic? (03:12:43 PM) ijchain: <dgp> the while loop condition (03:13:29 PM) _hc: the [info complete] while loop? (03:13:30 PM) ijchain: <dgp> First you test whether it's [info complete]. (03:13:47 PM) ijchain: <dgp> And then you test whether it ends in \n. (03:14:36 PM) ***ijchain ferrieux wonder whether [info complete $x\n] has been mentioned for the escaped-newline issue (03:14:40 PM) ijchain: <ferrieux> wonders 15:15 (03:15:00 PM) ijchain: <dgp> not mentioned. (03:15:08 PM) ijchain: <dgp> we can't even accomplish simple things. (03:15:13 PM) ijchain: <ferrieux> ah (03:15:26 PM) ijchain: <ferrieux> where's the current code ? (03:16:09 PM) ijchain: <dgp> _hc, I'd at least reverse the tests. (03:19:09 PM) ijchain: <dgp> you need [read -nonewline] ? 15:20 (03:21:01 PM) ijchain: <dgp> I give up; would be lots faster to just give you a working solution instead of tinker with what you have. _hc: aku kbk dgp thommey Gotisch: I think we have a winner, here's my new proc: http://pastebin.com/0ZzC4Vcz 03:31:52 PM) Gotisch: _hc: what happens if you get set a "1"; foreach b [c] { (03:32:00 PM) ijchain: <stu> Heh: (03:32:00 PM) ijchain: <stu> C == Slide Rule (03:32:00 PM) ijchain: <stu> C++ == Guy with a piece of paper in front of him and a pencil up his nose (03:32:04 PM) Gotisch: or better (03:32:08 PM) Gotisch: _hc: what happens if you get set a "1";\n foreach b [c] { (03:32:13 PM) Gotisch: \n isnt at end (03:32:18 PM) Gotisch: and command is not complete (03:33:21 PM) Gotisch: then again when foreach is finished it will work again (03:34:50 PM) ijchain: <aku> _hc - looks good 15:35 (03:35:24 PM) CIA-52: tclapps: Commit by patthoyts :: [Bug 3381002]: Applied patch from to fix the capabilities version string. (03:35:24 PM) CIA-52: apps/tkchat/ (ChangeLog tkchat.tcl) (03:36:41 PM) ***ijchain rz has become available (03:38:09 PM) _hc: Gotisch: I just ran a many megabytes of Tcl code through that proc, and no errors yet 15:40 (03:41:28 PM) ***ijchain ferrieux has left (03:41:30 PM) ***ijchain ferrieux has become available (03:42:54 PM) ijchain: <rz> Hi, do anyone have additional hints (beside the manual) about the cvs error: cvs [checkout aborted]: end of file from server (consult above messages if any) (03:43:26 PM) _hc: aku kbk dgp thommey Gotisch: thanks all so much! this proc seems noticeably faster, and has fixed the error so far! http://pastebin.com/0ZzC4Vcz (03:43:29 PM) ***ijchain pat has become available (03:43:50 PM) ijchain: <ferrieux> _hc, do you realize that your approach , picking an aligned \n every now and then, has a quadratic [info complete] test ? (03:44:18 PM) ijchain: <ferrieux> (I mean the sequence of calls is quadratic in the block size) (03:44:34 PM) _hc: I don't understand the quadtradic part (03:44:59 PM) ijchain: <ferrieux> you keep calling [info complete] on larger and larger blocks 15:45 (03:45:09 PM) ijchain: <ferrieux> until you get a simultaneously aligned and complete one (03:45:20 PM) ijchain: <ferrieux> right ? (03:45:45 PM) _hc: I ran a test, 'e' means executed the [read] call; "N" means no new line, wait for more data; "I" means failed [info complete], wait for new data; this is what it looked like at worst: eNNNNNNNNNNNNeNeeNNeNeeNeNeeNNeeNeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNeeNNeeNeNeNeNeNeNeNeeNNeeNeNeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNeeNNeeNeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeeNNeeNNeeNeNeNeeNeNeeNNeeNNeeNeNeNeNeNeNNeeNNeeNeNeNeeNeNeeNeeNNeeNeNeNeNeNeNeNeeNNeeNeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNeeNNeeNeNeNeNeNeNeNeeNNeeNeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeeNNeeNNeeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeNeNeeeNNeeNeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNeeNNeeNeNeNeNeNeNeNeeNNeeNeNeNeNeNeNeNeeNNeeNeeNeNeNeNeNeeNNeeNeNeeNeNeeeNeeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNNeNeeNeNNNNeNeNeNeNeeNNeeNeNeNeNeNeNeNeeNNeeNNeeNeNeNeNeNeeNNeeNNeeNeNeeNeNeNeeNNeeNeNeeNeeNeNeNeeNeeNeeNeNeeeee (03:45:52 PM) _hc: I never saw an I (03:46:04 PM) _hc: an "I" (03:46:13 PM) Zarutian [~zarutian@194-144-84-110.du.xdsl.is] entered the room. (03:46:19 PM) _hc: i.e. [info complete] is really just a fallback (03:46:21 PM) ***ijchain patthoyts has become available (03:46:30 PM) ijchain: <ferrieux> ok, on your data set it somehow works (03:47:03 PM) ijchain: <ferrieux> but [gets]+[info complete] would have done the job in amore robust way I think (03:47:08 PM) _hc: I think the vast majority of the time, there is a missing \n rather than more complicated missing things that info complete would catch (03:47:50 PM) _hc: [gets] reads line by line, I don't see how that wouldn't be slower than [read]'s block b block (03:48:42 PM) ***ijchain patthoyts has left (03:48:52 PM) ijchain: <ferrieux> it's all buffered on input, so the read() syscalls have the same granularity (03:49:11 PM) ijchain: <ferrieux> and the extra work on gets replaces the extra work on enormous [info complete]ß 15:50 (03:50:42 PM) ijchain: <ferrieux> but the trade-off depends on your data of course
On Jan 23, 2012, at 12:45 AM, Mathieu Bouchard wrote:
Le 2012-01-22 à 21:16:00, Miller Puckette a écrit :
There's not much way around this. One possibility (if indeed this is a serious efficiency issue) would be for Pd to append a "done" message to each batch up tcl to up-send.
That's called a newline... not preceded by a backslash. The thing with fconfigure -buffering line is that it doesn't care about backslashes, whereas eval does, and if you use both together, you need to account for that difference.
Switching to "fconfigure -buffering line" from "fconfigure -buffering none" did not change the size of chunks that were being executed. They were definitely larger than a single line, really many lines.
.hc
----------------------------------------------------------------------------
As we enjoy great advantages from inventions of others, we should be glad of an opportunity to serve others by any invention of ours; and this we should do freely and generously. - Benjamin Franklin
-
Hans-Christoph Steiner -
Mathieu Bouchard -
mescalinum＠gmail.com -
Miller Puckette