im wondering how others go about avoiding memory leaks caused by excessive symbol generation.
recently, ive found zexy-l2s to be the 'savior' to a lot of problems, such as:
.pool loses symbols embedded in lists after load/save .@list in toxy does not work with variable length lists .other objects work on symbols or float but not variable-length lists.. (or no way to toggle distribution to inlets) .qlist wont work with absolute timestamps..but oh wait, l2s & pipe does the trick! .want to insert 2, or maybe 3 or 4 elements, into a list after the first 2 elements .want to insert an entire list into another message, so use l2s and $1, etc...
everything works out but the catch is chew 1 mb RAM a minute or more, just from all the symbols being generated. im not sure i really need all them to stick around in some hash table after theyre handed off to another object or split up anyways..
ix wrote:
im wondering how others go about avoiding memory leaks caused by excessive symbol generation.
recently, ive found zexy-l2s to be the 'savior' to a lot of problems, such as:
.pool loses symbols embedded in lists after load/save .@list in toxy does not work with variable length lists .other objects work on symbols or float but not variable-length lists.. (or no way to toggle distribution to inlets) .qlist wont work with absolute timestamps..but oh wait, l2s & pipe does the trick! .want to insert 2, or maybe 3 or 4 elements, into a list after the first 2 elements .want to insert an entire list into another message, so use l2s and $1, etc...
everything works out but the catch is chew 1 mb RAM a minute or more, just from all the symbols being generated. im not sure i really need all them to stick around in some hash table after theyre handed off to another object or split up anyways..
i do not think that [l2s] should be ab-used to do complex list-operations (like inserting lists into lists,...) i mean, [ls2] generates a symbol and a symbol is meant to symbolize something rather than store data (well, forget that; but pd's strength is certainly not symbol-handling).
so we should improve pd's list handling rather than use symbol-handling for this. until this gets done, i would suggest to make some abstractions that emulate this handling; they will certainly be more efficient if you do not choose to convert numbers to strings and back again in between; and it will be more accurate, as pd's floats are real floats whereas converting them to symbols (strings) rounds them to 8 (or so) decimals.
<payed advertisment> [l], [packel], [niagara], [glue] (or some version of [prepend]), [length], [repack] will allow you to build(!) almost everything you need </payed advertisment>
mfg.as.r IOhannes
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://iem.at/cgi-bin/mailman/listinfo/pd-list
ix wrote:
.@list in toxy does not work with variable length lists
it does, since yesterday... I have finally decided to add the .#args parameter. Should I release toxy now, with the .wid file format slightly changed (supports a creator scriptlet), or wait until the format is likely to settle for good?
...
.qlist wont work with absolute timestamps..but oh wait, l2s & pipe does the trick!
or controlling a [qlist] with 'next', [t], [-], and [delay]?
.want to insert 2, or maybe 3 or 4 elements, into a list after the first 2 elements .want to insert an entire list into another message, so use l2s and $1, etc...
I entirely agree with Johannes about insertion being quite simple using zexy (or zl), and about the need for standardizing list handling into a built-in object (ala zl), or objects (ala zexy), with a carefully chosen small set of built-in features, to be extended with abstractions.
Krzysztof
On Nov 24, 2004, at 8:00 AM, Krzysztof Czaja wrote:
ix wrote:
.@list in toxy does not work with variable length lists
it does, since yesterday... I have finally decided to add the .#args parameter. Should I release toxy now, with the .wid file format slightly changed (supports a creator scriptlet), or wait until the format is likely to settle for good?
I am a fan of "release early, release often". I would just make it
clear that its alpha, and things could change without warning.
.hc
...
.qlist wont work with absolute timestamps..but oh wait, l2s & pipe
does the trick!or controlling a [qlist] with 'next', [t], [-], and [delay]?
.want to insert 2, or maybe 3 or 4 elements, into a list after the
first 2 elements .want to insert an entire list into another message, so use l2s and
$1, etc...I entirely agree with Johannes about insertion being quite simple using zexy (or zl), and about the need for standardizing list handling into a built-in object (ala zl), or objects (ala zexy), with a carefully chosen small set of built-in features, to be extended with abstractions.
Krzysztof
PD-list@iem.at mailing list UNSUBSCRIBE and account-management ->
http://iem.at/cgi-bin/mailman/listinfo/pd-list
"The arc of history bends towards justice."
Dr.
Martin Luther King, Jr.
moin ix, moin list,
On 24 November 2004 at 00:37:43, ix appears to have written:
im wondering how others go about avoiding memory leaks caused by excessive symbol generation.
recently, ive found zexy-l2s to be the 'savior' to a lot of problems, such as:
[snip]
everything works out but the catch is chew 1 mb RAM a minute or more, just from all the symbols being generated. im not sure i really need all them to stick around in some hash table after theyre handed off to another object or split up anyways..
Personally, I handle these memory leaks by not handling them: I use symbols pretty heavily in a number of patches (natural language stuff), I pre-define 30K or so symbols and generate more on the fly, and I agree that the permanence of symbols is a real difficulty on a machine with little RAM -- so far, I haven't managed to generate enough symbols to crash Pd during a performance though ;-)
Johannes and Krzysztof have commented on the specifics of l2s, and I can only add a hearty "me too" regarding some kind of standardized list handling primitives, which might help in the individual cases you're talking about. Another option (attractive to me) might be some kind of "temporary symbol" convention -- i.e. a symbol whose s_name would be changeable, but whose actual t_symbol* component is permanent -- I suppose such symbols would have to be identified by a sort of "initial name" to ensure that they get hashed right, but would have to be set-able in some non-standard fashion (list of ASCII-valued floats?) in order for them to make any sense memory-allocation-wise...
... or maybe i just think that sounds like a good idea because it's nearing 3am here....
marmosets, Bryan
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://iem.at/cgi-bin/mailman/listinfo/pd-list
On Thu, 25 Nov 2004, Bryan Jurish wrote:
Johannes and Krzysztof have commented on the specifics of l2s, and I can only add a hearty "me too" regarding some kind of standardized list handling primitives, which might help in the individual cases you're talking about. Another option (attractive to me) might be some kind of "temporary symbol" convention -- i.e. a symbol whose s_name would be changeable, but whose actual t_symbol* component is permanent -- I suppose such symbols would have to be identified by a sort of "initial name" to ensure that they get hashed right, but would have to be set-able in some non-standard fashion (list of ASCII-valued floats?) in order for them to make any sense memory-allocation-wise...
In LISP, which was the first language to have a Symbol data type, they added a type roughly like what you say, for the same reasons. They called it STRING. I think they had to import that feature from FORTRAN or BASIC.
I don't know when exactly they did add it, but a not-so-wild guess would be back then when "tty" still meant tele-typewriter, which would be sometime before I was born, and prolly even before you were born. I guess Scheme got strings from the beginning, and that was designed in the seventies, no?
Meanwhile it is still said that Pd doesn't deserve to get any more than symbols because Pd is not a string processing language. Well I don't mean to turn Pd into SED or SNOBOL, sorry, I just want strings.
BTW, is Pd going to get a refcounted-pointer atom type as in Python? or even mark-sweep-pointer atom type as in Ruby/ST/LISP ?
I'm imagining that, with the help of smarter pointers, all basic data types of Ruby/Python/Scheme/etc could become available to Pd more directly, and without leaks.
(One problem I'd have is, it seems that, in Pd, I never know whether a certain structure is supposed to be implemented as an atom-type, an object-class, a selector/method of a given object-class, or whatever. I feel disoriented.)
Mathieu Bouchard -=- Montréal QC Canada -=- http://artengine.ca/matju
moin Mathieu,
On 25 November 2004 at 07:18:59, Mathieu Bouchard wrote:
On Thu, 25 Nov 2004, Bryan Jurish wrote:
Another option (attractive to me) might be some kind of "temporary symbol" convention -- i.e. a symbol whose s_name would be changeable, but whose actual t_symbol* component is permanent -- I suppose such symbols would have to be identified by a sort of "initial name" to ensure that they get hashed right, but would have to be set-able in some non-standard fashion (list of ASCII-valued floats?) in order for them to make any sense memory-allocation-wise...
In LISP, which was the first language to have a Symbol data type, they added a type roughly like what you say, for the same reasons. They called it STRING. I think they had to import that feature from FORTRAN or BASIC.
yup, LISP was pretty much my inspiration for the idea; but by now, most LISPs have "real" strings and vectors anyways... PROLOG at least makes it look as if its strings are nothing more than lists of ASCII values, but I certainly don't want to turn Pd into PROLOG.
I don't know when exactly they did add it, but a not-so-wild guess would be back then when "tty" still meant tele-typewriter
... and when "car" really *did* mean "contents of the address register" ;-)
Meanwhile it is still said that Pd doesn't deserve to get any more than symbols because Pd is not a string processing language. Well I don't mean to turn Pd into SED or SNOBOL, sorry, I just want strings.
Likewise.
BTW, is Pd going to get a refcounted-pointer atom type as in Python? or even mark-sweep-pointer atom type as in Ruby/ST/LISP ?
I'm imagining that, with the help of smarter pointers, all basic data types of Ruby/Python/Scheme/etc could become available to Pd more directly, and without leaks.
I'm not sure that "smarter pointers" would really be a good way to go for Pd. If I understand your use of the term correctly, they would necessarily involve a garbage collection routine which could potentially get in the way of other computation -- ok, maybe it could be parked in its own thread, but that seems to me to be introducing too much complexity for relatively little gain: I'd be happy with some explicit control of the relevant memory management routines (allocate|set|copy|free) at the patch level, but maybe that's taking things too close to C...
(One problem I'd have is, it seems that, in Pd, I never know whether a certain structure is supposed to be implemented as an atom-type, an object-class, a selector/method of a given object-class, or whatever. I feel disoriented.)
As the Perl folks say, "TMTOWTDI". This is a Good Thing ;-)
marmosets, Bryan
On Thu, 25 Nov 2004, Bryan Jurish wrote:
On 25 November 2004 at 07:18:59, Mathieu Bouchard wrote:
In LISP, which was the first language to have a Symbol data type, they added a type roughly like what you say, for the same reasons. They called it STRING. I think they had to import that feature from FORTRAN or BASIC.
yup, LISP was pretty much my inspiration for the idea; but by now, most LISPs have "real" strings and vectors anyways...
Which is exactly what I am saying. Lisp's (format ...) function does the job of printf(), only it has several times more options. I'm saying that because LISP added Strings just after starting with Symbols, Pd should be the same; and then it would become in line not only with Lisp/Scheme/Dylan, but also with Smalltalk/Self/Ruby, and possibly a few more.
PROLOG at least makes it look as if its strings are nothing more than lists of ASCII values, but I certainly don't want to turn Pd into PROLOG.
Ok, so you suppose that strings are theoretically better served by the existing abstractions of list and integer? (i mean the "at least" part).
IMHO, I'd go more the Lisp way (if I'm not mistaken as to how it is solved.) There is a thing called a Sequence, which is not a class of data in particular, but is any class of data that can be trivially iterated: you can go through the elements in sequence. In other words, it's a "Virtual List" type. Vectors (the Lisp Arrays that have exactly 1 dimension of indices), the (linked) Lists, and Strings, are 3 examples of Sequences, and because of that they have a lot of operations in common.
If I misread that about Lisp, then there is the same in Smalltalk and Ruby anyway. In the latter it's a pseudo-class called Enumerable and contains methods for making various kinds of for-loops on the elements.
... and when "car" really *did* mean "contents of the address register" ;-)
Sigh yes. Nowadays it sounds so silly that it gets deserved puns such as "My other car is a cdr".
I'm not sure that "smarter pointers" would really be a good way to go for Pd. If I understand your use of the term correctly, they would necessarily involve a garbage collection routine which could potentially get in the way of other computation --
With reference counting, the only potential spike is the time that free() may take on a bad day, but that's nothing on the same scale as the main thread blocking due to socket abuse wrt to the GUI.
However with mark'n'sweep, there is a small delay (i think it's 25 ms nowadays on a typical setup of Ruby/GridFlow/Pd) that is caused by non-delayed garbage collection. In Ruby you can disable GC for a short while, which could be useful, if the audio didn't have a need for continuous CPU, hence the solution to that is either a faster GC, a multi-pass GC, or a threaded GC.
ok, maybe it could be parked in its own thread, but that seems to me to be introducing too much complexity for relatively little gain:
For Ruby itself it will make sense in the future and then GridFlow will behave properly on the audio side (I mean the latency will be reducible). For Pd it's overkill, and since the only simple mark+sweep is non-realtime, we're stuck with ref-counting. But that's not such a big problem because Perl and Python still use ref-counting, and the problems you get with circular references with not too much hassle; techniques include:
Weak Pointers, which are pointers that do not affect the ref-count, and self-destruct when the pointed-to is gone;
Symbolic/Numeric References: you instead record the location of an object in terms of another pointer or global name, and then also you record how to get to your object from there. e.g. the list "myarray1 42" can be used to talk about element #42 of the array myarray1 declared globally, without having a true pointer to it.
I'd be happy with some explicit control of the relevant memory management routines (allocate|set|copy|free) at the patch level, but maybe that's taking things too close to C...
That _is_ too close to C. People can scream as hard as they want about GridFlow being conceptually hard, but inviting people to allocate and free stuff themselves is pure suicide and a real joke.
(One problem I'd have is, it seems that, in Pd, I never know whether a certain structure is supposed to be implemented as an atom-type, an object-class, a selector/method of a given object-class, or whatever. I feel disoriented.)
As the Perl folks say, "TMTOWTDI". This is a Good Thing ;-)
Being well-versed in Perl/Ruby, i know the mentality. (In Ruby, there are nine different syntactic forms for an "if" statement. Really.) However in any language, as minimal as you want (Scheme), there are also always several ways to do anything. The problem is more about exactly how many (and which ones) you should provide to make the users comfortable.
Well, the issue I wanted to raise is slightly different. It has to do with the mapping of concepts from one language (whichever) to the other (here, PureData). Very often, something that is a class in Pd is normally just a method outside of Pd (like [+]), and sometimes it's the other way, like [f] being a holder of float (and also a message type and an atom type, which are very distinct things!).
I would enjoy some kinds of guidelines for the (potentially automatic but possibly instead manual and heuristic) mapping of functionality from a programming language like Perl/Python/Ruby/Scheme to PureData.
Mathieu Bouchard -=- Montréal QC Canada -=- http://artengine.ca/matju
-
Bryan Jurish -
Hans-Christoph Steiner -
ix -
Johannes M Zmoelnig -
Krzysztof Czaja -
Mathieu Bouchard -
Thomas Grill