Hey all,
I've posted to the list before about getting tcl errors in some of my new patches. Well after more digging I have finally found the cause.
I'll first explain the patch a bit. tcl_errors.pd is the toplevel patch. It contains a number of dir2abstractionArray abstractions. These are abstractions that create dynamic subpatches of abstractions. The abstractions are read from a directory you specify via set-dir messages. A TOT object reads the directory and then spits out a list of all the abstractions. Each one is then dripped and has the extension removed. This is sent to the dynamic patch generator that actually creates the "array" of abstractions in a subpatch. The dir2abstractionArray abstraction brokers the communication from the outside to the various abstractions contained in the array. The first two inlets and outlets get routed to the "selected" abstraction. The selected abstraction is chosen by index value.
I'm using this abstraction so that an end-user can drop custom abstractions into special directories and they automatically get updated in the GUI when the patch is loaded. This could even happen while PD is running by sending an update message.
Due to the complexity of the GOP that uses this abstraction I had a lot of trouble seeing where the tcl errors were coming from.
When we have only one dir2abstractionArray in a patch then it behaves very well (on OSX) on linux if you try and update the array too fast you get tcl errors. On both linux and OSX if we have more than one instance of this abstraction then all hell breaks loose and we get constant tcl errors. :(
The most obvious solution is to send the update (pattern) messages in sequence and not all at once. The problem is that the solution needs to work in every pixelTANGO patch, no matter how many abstractions there are or what order they are in. There is no way of picking out a particular abstraction to go first. How do the others know when its done? I had been using delays, but since all the abstractions are the same, they all get delayed for the same amount of time. I guess I could divide some constant by the $0 variable to get a unique delay for each abstraction. Does that make sense??
I'm not sure if this would be considered a real bug, since its so bleeding edge and perhaps is not supposed to work at all..
Krzysztof, are the tcl errors happening because I'm running simultaneous "query" commands that are being bogged down by the dynamic patching?
I'm doing a workshop on thursday night and I would really like to get this fixed, I feel like I'm very close since I can actually make a test-patch. I don't have any brilliant ideas about how to fix it though.
Using the test patch:
1. Make sure tot and zexy are working 2. save all the patches in one directory and then open the tcl_errors.pd file. 3. Change the set-dir arguments to a directory on your machine that is in your PD path. (better NOT choose the directory containing the tcl_errors patch) or try the sending "sel-dir" message. 4. Try the "pattern *.pd" message at the top and look for tcl errors. If you click it slowly enough you should get no tcl errors. 5. Try the "pattern *.pd" message attached to multiple abstractions. This will cause tcl errors every time.
A patch fix would be good for now, but it would be nice if these errors could somehow be avoided in the future (perhaps with changes in pd and or TOT?)
Thank you all for your time. I took me months to get this far, I hope there are some brilliant ideas to fix things out there.
B. Bogart
#N canvas 137 64 789 679 10; #X obj 362 408 + 10; #X obj 365 363 t f f; #X obj 392 341 + 1; #X obj 336 318 t a b; #X obj 360 341 f 0; #X msg 382 318 0; #X text 435 222 reset; #X obj 409 437 f $0; #X obj 362 387 * 120; #X obj 437 241 loadbang; #X obj 313 275 stripExtension; #X obj 313 254 drip; #X obj 18 167 inlet; #X obj 28 190 s $0-in; #X obj 28 403 r $0-out; #X obj 18 426 outlet; #X obj 241 115 inlet; #X obj 449 315 makesymbol pd-%s-array; #X obj 449 295 f $0; #N canvas 198 331 810 498 1001-array 0; #X restore 19 455 pd $0-array; #X obj 313 158 t a b; #X obj 464 178 s $0-selector; #X obj 446 149 inlet; #X obj 525 441 outlet; #X text 18 120 which Abst to route through?; #X text 528 459 List of options; #X obj 525 421 prepend Passthrough; #X obj 18 373 spigot; #X msg 363 182 1; #X obj 430 178 == 0; #X obj 382 437 + 1; #X obj 357 254 outlet; #X obj 106 167 inlet; #X obj 106 426 outlet; #X obj 106 373 spigot; #X obj 116 190 s $0-in2; #X obj 116 403 r $0-out2; #X obj 255 254 length; #X obj 263 275 outlet; #X obj 241 531 makesymbol pd-%s-array; #X msg 241 551 ; $1 clear; #X obj 241 511 f $0; #X obj 241 186 route clear; #X obj 336 470 pack s f f f s f; #X obj 443 437 f $1; #X obj 438 263 t b b b; #X msg 336 496 ; $5 obj $2 10 $1 $3 $4 $6; #X obj 255 345 - 1; #X obj 268 395 >=; #X obj 267 418 select 1; #X obj 542 571 outlet; #X text 541 591 Output bang when done; #X text 14 10 This abstraction takes a glob pattern in a directory and creates an array of the abstractions in that folder in an internal subpatch called [pd $0-array]. The abstractions are meant to be routing objects so they all have a global receive and send , but only one abstraction in the array at a time will receive input. The abstractions must contain at least a arrayRouterIn abstraction to broker the communication. ; #X obj 313 210 dirlist; #X connect 0 0 43 1; #X connect 1 0 8 0; #X connect 1 1 30 0; #X connect 2 0 4 1; #X connect 3 0 43 0; #X connect 3 1 4 0; #X connect 4 0 2 0; #X connect 4 0 1 0; #X connect 4 0 48 0; #X connect 5 0 4 1; #X connect 7 0 43 3; #X connect 8 0 0 0; #X connect 9 0 5 0; #X connect 9 0 28 0; #X connect 9 0 45 0; #X connect 10 0 3 0; #X connect 11 0 10 0; #X connect 12 0 13 0; #X connect 12 0 27 0; #X connect 14 0 15 0; #X connect 16 0 20 0; #X connect 17 0 43 4; #X connect 18 0 17 0; #X connect 20 0 42 0; #X connect 20 1 18 0; #X connect 20 1 5 0; #X connect 20 1 28 0; #X connect 22 0 21 0; #X connect 22 0 29 0; #X connect 26 0 23 0; #X connect 27 0 15 0; #X connect 28 0 27 1; #X connect 28 0 34 1; #X connect 29 0 27 1; #X connect 29 0 34 1; #X connect 30 0 43 2; #X connect 32 0 34 0; #X connect 32 0 35 0; #X connect 34 0 33 0; #X connect 36 0 33 0; #X connect 37 0 38 0; #X connect 37 0 47 0; #X connect 39 0 40 0; #X connect 41 0 39 0; #X connect 42 0 41 0; #X connect 42 1 53 0; #X connect 43 0 46 0; #X connect 44 0 43 5; #X connect 45 0 7 0; #X connect 45 1 18 0; #X connect 45 2 44 0; #X connect 47 0 48 1; #X connect 48 0 49 0; #X connect 49 0 50 0; #X connect 53 0 11 0; #X connect 53 0 26 0; #X connect 53 0 37 0; #X connect 53 1 31 0;
#N canvas 87 227 619 468 10; #X obj 20 210 tot .; #X obj 86 107 dirpanel; #X obj 20 61 inlet; #X text 26 302 Can't read dirs with spaces... how to wrap??; #X obj 20 234 outlet; #X text 20 17 Lists the files in a directory thanks to dirpanel. Sends the files matching pattern as a list.; #X obj 20 110 symbol; #X text 26 271 ONLY sends absolute pathnames , otherwise hard to add before load! No internel data-structure to exploit.; #X obj 103 235 outlet; #X text 154 236 Output chosen directory; #X obj 20 84 route pattern sel-dir set-dir; #X obj 231 84 loadbang; #X obj 64 158 pack s s; #X text 129 200 $1 is pattern $2 is dir; #X msg 63 180 query glob [file join $2 $1]; #X obj 86 129 symbol; #X obj 152 107 symbol; #X obj 117 352 print tot; #N canvas 0 22 462 312 pwd 0; #X text 18 27 This is where used to be.; #X obj 35 92 inlet; #X obj 41 184 outlet; #X obj 48 153 symbol /; #X connect 1 0 3 0; #X connect 3 0 2 0; #X restore 231 107 pd pwd; #X connect 0 0 4 0; #X connect 1 0 15 0; #X connect 2 0 10 0; #X connect 6 0 12 0; #X connect 10 0 6 0; #X connect 10 1 1 0; #X connect 10 2 16 0; #X connect 10 2 8 0; #X connect 11 0 18 0; #X connect 12 0 14 0; #X connect 14 0 0 0; #X connect 14 0 17 0; #X connect 15 0 12 1; #X connect 15 0 8 0; #X connect 16 0 12 1; #X connect 18 0 12 1;
#N canvas 500 454 458 308 10; #X obj 39 130 tot .; #X msg 39 105 query tk_chooseDirectory; #X obj 39 54 inlet; #X obj 39 78 bang; #X obj 39 155 tosymbol; #X obj 39 180 outlet; #X text 16 9 Sends the pathname of a directory on bang; #X connect 0 0 4 0; #X connect 1 0 0 0; #X connect 2 0 3 0; #X connect 3 0 1 0; #X connect 4 0 5 0;
#N canvas 0 22 454 304 10; #X obj 15 99 tot .; #X msg 15 77 query file rootname $1; #X obj 15 55 symbol; #X obj 15 33 inlet; #X obj 15 123 outlet; #X text 4 5 Uses toxy to remove the extension from a symbol (.*); #X connect 0 0 4 0; #X connect 1 0 0 0; #X connect 2 0 1 0; #X connect 3 0 2 0;
#N canvas 442 118 802 582 10; #X obj 26 315 dir2abstractionArray $0; #X text 21 16 Requires TOT and ZEXY; #X obj 152 401 dir2abstractionArray $0; #X obj 89 354 dir2abstractionArray $0; #X obj 391 490 dir2abstractionArray $0; #X obj 313 462 dir2abstractionArray $0; #X obj 229 435 dir2abstractionArray $0; #X obj 23 138 dir2abstractionArray $0; #X msg 127 95 pattern *.pd; #X msg 317 230 pattern *.pd; #X text 490 183 <- Same directory as above; #X msg 362 255 clear; #X text 418 255 <- to clear dynamic patch; #X obj 225 96 cnv 15 70 15 empty empty empty 20 12 0 14 -257281 -66577 0; #X obj 418 232 cnv 15 70 15 empty empty empty 20 12 0 14 -257281 -66577 0; #X text 344 60 1 Choose a directory containing your abstractions.; #X text 227 96 <- WORKS: when you create the array no tcl errors.; #X text 417 232 <- ERRORS: with multiple abstractions; #X msg 87 61 set-dir /store/temp/debug/tcl_errors; #X msg 219 185 set-dir /store/temp/debug/tcl_errors; #X text 298 116 On linux simply clicking too fast causes tcl errors ; #X connect 8 0 7 2; #X connect 9 0 0 2; #X connect 9 0 2 2; #X connect 9 0 3 2; #X connect 9 0 4 2; #X connect 9 0 5 2; #X connect 9 0 6 2; #X connect 11 0 0 2; #X connect 11 0 3 2; #X connect 11 0 2 2; #X connect 11 0 6 2; #X connect 11 0 5 2; #X connect 11 0 4 2; #X connect 18 0 7 2; #X connect 19 0 0 2; #X connect 19 0 3 2; #X connect 19 0 2 2; #X connect 19 0 6 2; #X connect 19 0 5 2; #X connect 19 0 4 2;
hi Ben,
B. Bogart wrote: ...
I'm not sure if this would be considered a real bug, since its so bleeding edge and perhaps is not supposed to work at all..
Krzysztof, are the tcl errors happening because I'm running simultaneous "query" commands that are being bogged down by the dynamic patching?
...
#X msg 63 180 query glob [file join $2 $1];
...
#X msg 15 77 query file rootname $1;
tot is not, and never was meant to be a scripting tool. It is a simple little hack for talking with the gui, in a gentle manner.
You cannot ever pull more than 4Kb down to pd from pd-gui. If you send multiple queries, the responses are appended to an internal Pd buffer of 4Kb size. Under heavy load that buffer is bound to overflow before being served.
Choose any proper scripting external or shell, and replace all tots.
Besides, even without tots, if the abstractions you expect people will store in the set-dir directory have open windows and such, it is begging for trouble sooner or later. Pd's communication to and from pd-gui is not robust enough for that kind of things.
Krzysztof
Thank you Krzysztof!
Finally I know why! :) I do realize I was playing with the bleeding edge here, but tot seemed much more accessible for simple things than pyext (since I don't know python!)
Has pyext emerged as the standard cross-platform way of doing scripting in PD? (directory listing, simple string processing etc..) I realize I could use shell, but would rather avoid future difficulties in windows now. One question about pyext, can a single external pass data internally to another instance? That is can multiple pyext scripts share data via python rather than through the patch?
I don't see the issue with open patches since someone with a patch open in one window and calling it as an abstraction in another would have the same problem as my system would? Its been a while since I did anything complex on linux. On OSX a few strange things happen with you save abstractions contained in other abstractions. There is a tendancy for PD to open the abstractions that contain the abstraction that was just saved. Ugly with many copies of a single abstraction. If people are not aware of this one I'll make a little demo patch.
I am guessing it is in all our best interests that the PD gui communication does become more robust!
Thanks again Krzysztof for poiting out my design error!!
B.
Czaja wrote:
hi Ben,
B. Bogart wrote: ...
I'm not sure if this would be considered a real bug, since its so bleeding edge and perhaps is not supposed to work at all..
Krzysztof, are the tcl errors happening because I'm running simultaneous "query" commands that are being bogged down by the dynamic patching?
...
#X msg 63 180 query glob [file join $2 $1];
...
#X msg 15 77 query file rootname $1;
tot is not, and never was meant to be a scripting tool. It is a simple little hack for talking with the gui, in a gentle manner.
You cannot ever pull more than 4Kb down to pd from pd-gui. If you send multiple queries, the responses are appended to an internal Pd buffer of 4Kb size. Under heavy load that buffer is bound to overflow before being served.
Choose any proper scripting external or shell, and replace all tots.
Besides, even without tots, if the abstractions you expect people will store in the set-dir directory have open windows and such, it is begging for trouble sooner or later. Pd's communication to and from pd-gui is not robust enough for that kind of things.
Krzysztof
PD-dev mailing list PD-dev@iem.at http://iem.at/cgi-bin/mailman/listinfo/pd-dev
Hi Ben,
Has pyext emerged as the standard cross-platform way of doing scripting in PD? (directory listing, simple string processing etc..) I realize I could use shell, but would rather avoid future difficulties in windows now. One question about pyext, can a single external pass data internally to another instance? That is can multiple pyext scripts share data via python rather than through the patch?
i don't know if py/pyext is some kind of standard... i'm still not sure either if it's lightweight enough for usage in a real-time system, although i'm constantly trying to improve the situation. I think that the "detached modes" (1 and 2) should work pretty well in the meantime. Currently, all py/pyext objects share a common interpreter, but each script is loaded as a discrete module. In this sense equal scripts (loadable by several py/pyext objects) can share "global" data, whereas different script should (i never tried this) be able to share data within modules they "imported". In the future i plan some even more detached mode, where the whole script (including initialization) is running in the background, if desired every instance with its own interpreter.
best greetings, Thomas