hi all, i am trying to develop a string external that does not use symbols for string storage because of known problems with symbols.
what i am now doing is using string handles (int) and store the strings internally in my external. that works quite well. the only problem is that i had to introduce usage counters to free my string memory when it is no longer needed. when a string is generated (each operation that returns a string generates a new string) the usage count is set to one and when an object receives a string at one of its inlets the usage count is reduced. if the usage count is zero the string mem is freed. that works as long as i don't split outlets. my objects don't know how many connections are plugged into an outlet so it cannot set the usage count to the number of connections as it should be. so my (not very beautiful) solution is to introduce an object that splits an inlet into multiple outlets and increases the usage counter accordingly.
i did not find a possibility to find out the number of connections attached to an outlet and i guess there is no method for that. is that correct? or could maybe future versions of pd make it possible for externals to detect the number of connections?
anyway my solution works quite well but i'd like to improve it.
best joerg
moin joerg,
On 13 October 2004 at 14:08:55, joerg piringer appears to have written:
hi all, i am trying to develop a string external that does not use symbols for string storage because of known problems with symbols.
sounds excellent -- i have been chewing on string represntations for pd for a while; my only real current implementation is the 'pd-string' package (http://www.ling.uni-potsdam.de/~moocow/projects), which has 2 objects: [any2string] and [string2any] -- "strings" are passed around as lists of floats (in the range 0..255) -- it's pretty space-inefficient, but it does appear to work.
what i am now doing is using string handles (int) and store the strings internally in my external. that works quite well. the only problem is that i had to introduce usage counters to free my string memory when it is no longer needed.
[snip]
... that sounds pretty complex to me. perhaps we might work together to crunch more string data into a "list", maybe using a special selector ('string'? 'ascii'?) for it? that way, we could avoid at least some hairiness with reference-counting: it looks like at least 3 bytes can be encoded losslessly in one float (for later extraction with bitwise ops), which is still wasting 1/4 of the allocated space, but I think I can live with that... you?
marmosets, Bryan
what i am now doing is using string handles (int) and store the strings
pd doesn't have a buildin int type ... just a suggestion, but what about (ab)using the pointer typecasted to floats or the pointer in t_symbol ...
cheers ... t
i don't want to use pointers because if i send an invalid pointer to one of my string objects it will most likely crash pd. therefore i use handles. the worst thing that can happen is that a wrong handle refers to the wrong string. i made tests with wrong handles and it is quite fail safe.
i also don't want to use lists because my string class is based on cstring (because it shares code with a "text-processor" program i am writing for my pda) and so i'd have to do a lot of converting between list and string and back again just to do one operation. that cannot be effective. i also don't know how pd handles lists, does it copy a list when i connect two objects to one outlet? or reference counting?
best joerg
Tim Blechmann wrote:
what i am now doing is using string handles (int) and store the strings
pd doesn't have a buildin int type ... just a suggestion, but what about (ab)using the pointer typecasted to floats or the pointer in t_symbol ...
cheers ... t
i don't want to use pointers because if i send an invalid pointer to one of my string objects it will most likely crash pd. therefore i use handles. the worst thing that can happen is that a wrong handle refers to the wrong string. i made tests with wrong handles and it is quite fail safe.
you could store the pointers in a list and test, if it's valid ... the problem with ints that are represented by floats ... you will have a point when (x+1 == x) returns true ... which might be a wonderful source for errors ...
cheers ... tim
Tim Blechmann wrote:
i don't want to use pointers because if i send an invalid pointer to one of my string objects it will most likely crash pd. therefore i use handles. the worst thing that can happen is that a wrong handle refers to the wrong string. i made tests with wrong handles and it is quite fail safe.
you could store the pointers in a list and test, if it's valid ...
yes, that's what i call a handle.
the problem with ints that are represented by floats ... you will have a point when (x+1 == x) returns true ... which might be a wonderful source for errors ...
thanks for the hint. i'll check that out.
j
On Wed, 13 Oct 2004, Tim Blechmann wrote:
i don't want to use pointers because if i send an invalid pointer to one of my string objects it will most likely crash pd. therefore i use handles. the worst thing that can happen is that a wrong handle refers to the wrong string. i made tests with wrong handles and it is quite fail safe.
you could store the pointers in a list and test, if it's valid ... the problem with ints that are represented by floats ... you will have a point when (x+1 == x) returns true ... which might be a wonderful source for errors ...
That number is 1<<24, that is, 16777216, for the float32 type. If you want to pass anything else with more bits, the trick is that, if you have a int x and a float y, instead of doing x=y you'd do *(int *)x = y;
but don't expect things pointers converted to text and back to work, even if within the 1<<24 range, because of binary-to-decimal conversion, which by itself isn't so bad, but Pd chops off extra digits.
____________________________________________________________________ Matsjö Buschahr @ Ruby Konferenz, Muenchen http://artengine.ca/matju
moin Joerg, moin Tim, moin List,
just some thoughts:
On 13 October 2004 at 16:43:36, joerg piringer appears to have written:
i don't want to use pointers because if i send an invalid pointer to one of my string objects it will most likely crash pd.
... but that's basically what you're doing now (with additional reference counting), or did I misunderstand?
therefore i use handles. the worst thing that can happen is that a wrong handle refers to the wrong string. i made tests with wrong handles and it is quite fail safe.
... or are you actually adding another table-lookup: it ought to be straightforward enough to use literal pointer values (maybe as Tim suggested by (ab)using w_symbol.s_name) and just keep an ID-hash of "valid" ones around, pointing to their reference counts. But if you go this route, you're doing some extra computation anyways...
i also don't want to use lists because my string class is based on cstring (because it shares code with a "text-processor" program i am writing for my pda) and so i'd have to do a lot of converting between list and string and back again just to do one operation. that cannot be effective.
it's a bit of processing overhead, that's certainly true; and my existing message<->list conversion routines are pretty ugly at the moment (lots of unnecessary re-allocation), but I haven't noticed them slowing me down, even in pretty hefty usage ... of course, I deal mostly with many rapidly changing short strings (English words), and if you're dealing with large chunks of data, your mileage may vary...
conversion would also be additional code to call, but if we have a general convention for string-handling, this could conceivably be packed into a library and (statically) linked in, right?
i also don't know how pd handles lists, does it copy a list when i connect two objects to one outlet? or reference counting?
none of the above, as far as i know: my impression has always been (at least, since some previous hypotheses caused segfaults ;-) that the "originator" (to borrow a term from Frank) of a multi-atom message (i.e. a list) is responsible for allocating and maintaining the memory associated with that list, which can be safely freed when the object itself disappears. So basically, whenever message data gets changed by passing through an object, the object should copy that data into a local buffer, and pass the (modified) local copy back out -- again probably not what you want if you're dealing with large chunks of shared data: maybe pdp would be the way to go in that case.
marmosets, Bryan
Bryan Jurish wrote:
moin Joerg, moin Tim, moin List,
just some thoughts:
On 13 October 2004 at 16:43:36, joerg piringer appears to have written:
i don't want to use pointers because if i send an invalid pointer to one of my string objects it will most likely crash pd.
... but that's basically what you're doing now (with additional reference counting), or did I misunderstand?
no, i have an array with indexes. each string handle is an index in my array.
therefore i use handles. the worst thing that can happen is that a wrong handle refers to the wrong string. i made tests with wrong handles and it is quite fail safe.
... or are you actually adding another table-lookup: it ought to be straightforward enough to use literal pointer values (maybe as Tim suggested by (ab)using w_symbol.s_name) and just keep an ID-hash of "valid" ones around, pointing to their reference counts. But if you go this route, you're doing some extra computation anyways...
i am using (at the moment) a fixed array of pointers to stringobjects that can be allocated. each handle is an index of one of these stringobjects. when i free a stringobject the handle is stored in a free-handle-list. so new allocations are made for these second hand handles first.
i also don't want to use lists because my string class is based on cstring (because it shares code with a "text-processor" program i am writing for my pda) and so i'd have to do a lot of converting between list and string and back again just to do one operation. that cannot be effective.
it's a bit of processing overhead, that's certainly true; and my existing message<->list conversion routines are pretty ugly at the moment (lots of unnecessary re-allocation), but I haven't noticed them slowing me down, even in pretty hefty usage ... of course, I deal mostly with many rapidly changing short strings (English words), and if you're dealing with large chunks of data, your mileage may vary...
i am dealing with large strings, that's why i want to keep things fast.
i also don't know how pd handles lists, does it copy a list when i connect two objects to one outlet? or reference counting?
none of the above, as far as i know: my impression has always been (at least, since some previous hypotheses caused segfaults ;-) that the "originator" (to borrow a term from Frank) of a multi-atom message (i.e. a list) is responsible for allocating and maintaining the memory associated with that list, which can be safely freed when the object itself disappears. So basically, whenever message data gets changed by passing through an object, the object should copy that data into a local buffer, and pass the (modified) local copy back out -- again probably not what you want if you're dealing with large chunks of shared data: maybe pdp would be the way to go in that case.
pdp: no, i am using evil winxp.
i think my concept works quite well. i will experiment a little bit more...
best joerg
On Wed, 13 Oct 2004, joerg piringer wrote:
when a string is generated (each operation that returns a string generates a new string) the usage count is set to one and when an object receives a string at one of its inlets the usage count is reduced. if the usage count is zero the string mem is freed.
Why can't you just free the string after it has been sent ? AFAIK the outlet_* functions execute the methods of the connected objects immediately.
Guenter
that works as long as i don't split outlets. my objects don't know how many connections are plugged into an outlet so it cannot set the usage count to the number of connections as it should be. so my (not very beautiful) solution is to introduce an object that splits an inlet into multiple outlets and increases the usage counter accordingly.
i did not find a possibility to find out the number of connections attached to an outlet and i guess there is no method for that. is that correct? or could maybe future versions of pd make it possible for externals to detect the number of connections?
anyway my solution works quite well but i'd like to improve it.
best joerg
-- http://joerg.piringer.net http://www.transacoustic-research.com http://www.iftaf.org http://www.vegetableorchester.org
PD-dev mailing list PD-dev@iem.at http://iem.at/cgi-bin/mailman/listinfo/pd-dev
guenter geiger wrote:
On Wed, 13 Oct 2004, joerg piringer wrote:
when a string is generated (each operation that returns a string generates a new string) the usage count is set to one and when an object receives a string at one of its inlets the usage count is reduced. if the usage count is zero the string mem is freed.
Why can't you just free the string after it has been sent ? AFAIK the outlet_* functions execute the methods of the connected objects immediately.
ah. interesting. but what happens if i want to store the string in a list, or use a builtin pd object?
joerg
On Wed, 13 Oct 2004, joerg piringer wrote:
Why can't you just free the string after it has been sent ? AFAIK the outlet_* functions execute the methods of the connected objects immediately.
ah. interesting. but what happens if i want to store the string in a list, or use a builtin pd object?
It probably won't work if you want to store, delay and such things with standard objects, using float. But then, a string should not be an int, or should it ?
Guenter
-
Bryan Jurish -
guenter geiger -
joerg piringer -
Matju -
Tim Blechmann