Hi,
I have deleted Miller's reply where he said that he's not that interested in adding a string type to Pd, but I'd like to ask him a couple of questions regarding that response, if that's ok.
1. How do you propose to solve the 'spaces in file path' issue without a string type? Or are you content with that restriction?
2. How do you suggest that people deal with the symbol table pollution issue mentioned before on this list, when they are doing operations processing lots and lots of symbol-strings in Pd? Let me know if you want more information about this issue.
3. Will a [symbol2list] ever make it into Pd canonical so that people can split long symbols on a character, like the zexy external that does this? It seems strange that you can concatenate symbols, but not split them apart again.
4. Can anyone else help me with a concise summary of other string/Pd issues I haven't thought of?
Thanks for taking the time to read and reply.
Best,
Chris.
------------------- http://mccormick.cx
HI all,
I don't have answers to all these, but I'm sure that adding a string ytpe to Pd isn't the roght way to handle these problems. But specifically:
1. spaces in symbols are a parsing/formatting problem, not a data type problem; 2. use arrays as strings as I proposed; 3. I have to think about that one some more (!) and 4. one thing is dealing with non-ascii character sets, although there are likely to be many more problems to adress.
On Tue, Nov 13, 2007 at 11:57:06PM -0500, Chris McCormick wrote:
Hi,
I have deleted Miller's reply where he said that he's not that interested in adding a string type to Pd, but I'd like to ask him a couple of questions regarding that response, if that's ok.
- How do you propose to solve the 'spaces in file path' issue without a
string type? Or are you content with that restriction?
- How do you suggest that people deal with the symbol table pollution
issue mentioned before on this list, when they are doing operations processing lots and lots of symbol-strings in Pd? Let me know if you want more information about this issue.
- Will a [symbol2list] ever make it into Pd canonical so that people
can split long symbols on a character, like the zexy external that does this? It seems strange that you can concatenate symbols, but not split them apart again.
- Can anyone else help me with a concise summary of other string/Pd
issues I haven't thought of?
Thanks for taking the time to read and reply.
Best,
Chris.
PD-dev mailing list PD-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
Using arrays as strings is an interesting idea. I don't think non- ascii charsets should be too big a deal, they are decently supported right now, without even trying :). The Pd floats should store UTF-16 fine, which really covers basically everything. By the time UTF-32 is used much, Pd will be using 64-bit floats.
.hc
On Nov 14, 2007, at 12:43 AM, Miller Puckette wrote:
HI all,
I don't have answers to all these, but I'm sure that adding a string ytpe to Pd isn't the roght way to handle these problems. But specifically:
- spaces in symbols are a parsing/formatting problem, not a data
type problem; 2. use arrays as strings as I proposed; 3. I have to think about that one some more (!) and 4. one thing is dealing with non-ascii character sets, although there are likely to be many more problems to adress.
On Tue, Nov 13, 2007 at 11:57:06PM -0500, Chris McCormick wrote:
Hi,
I have deleted Miller's reply where he said that he's not that interested in adding a string type to Pd, but I'd like to ask him a couple of questions regarding that response, if that's ok.
- How do you propose to solve the 'spaces in file path' issue
without a string type? Or are you content with that restriction?
- How do you suggest that people deal with the symbol table
pollution issue mentioned before on this list, when they are doing operations processing lots and lots of symbol-strings in Pd? Let me know if you want more information about this issue.
- Will a [symbol2list] ever make it into Pd canonical so that people
can split long symbols on a character, like the zexy external that does this? It seems strange that you can concatenate symbols, but not split them apart again.
- Can anyone else help me with a concise summary of other string/Pd
issues I haven't thought of?
Thanks for taking the time to read and reply.
Best,
Chris.
PD-dev mailing list PD-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
PD-dev mailing list PD-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
------------------------------------------------------------------------ ----
"Free software means you control what your computer does. Non-free software means someone else controls that, and to some extent controls you." - Richard M. Stallman
morning all,
One (potential) problem with strings-as-arrays (as has been pointed out before) is a gross waste of space: for IEEE-754 floats, sizeof(float)=4, which is 3 bytes wasted for a single 8-bit character. We could do some devious byte-packing and pointer-casting in one or more externals to attenuate the problems, but that gets very cryptic very quickly (I actually tried this with [pdstring] (which wastes even *more* memory by using one atom per character), but decided against it in the interests of clarity and flexibility)... I think it's definitely worth a go however...
marmosets, Bryan
On 2007-11-14 08:13:03, Hans-Christoph Steiner hans@eds.org appears to have written:
Using arrays as strings is an interesting idea. I don't think non- ascii charsets should be too big a deal, they are decently supported right now, without even trying :). The Pd floats should store UTF-16 fine, which really covers basically everything. By the time UTF-32 is used much, Pd will be using 64-bit floats.
.hc
On Nov 14, 2007, at 12:43 AM, Miller Puckette wrote:
HI all,
I don't have answers to all these, but I'm sure that adding a string ytpe to Pd isn't the roght way to handle these problems. But specifically:
- spaces in symbols are a parsing/formatting problem, not a data
type problem; 2. use arrays as strings as I proposed; 3. I have to think about that one some more (!) and 4. one thing is dealing with non-ascii character sets, although there are likely to be many more problems to adress.
On Tue, Nov 13, 2007 at 11:57:06PM -0500, Chris McCormick wrote:
Hi,
I have deleted Miller's reply where he said that he's not that interested in adding a string type to Pd, but I'd like to ask him a couple of questions regarding that response, if that's ok.
- How do you propose to solve the 'spaces in file path' issue
without a string type? Or are you content with that restriction?
- How do you suggest that people deal with the symbol table
pollution issue mentioned before on this list, when they are doing operations processing lots and lots of symbol-strings in Pd? Let me know if you want more information about this issue.
- Will a [symbol2list] ever make it into Pd canonical so that people
can split long symbols on a character, like the zexy external that does this? It seems strange that you can concatenate symbols, but not split them apart again.
- Can anyone else help me with a concise summary of other string/Pd
issues I haven't thought of?
Thanks for taking the time to read and reply.
Best,
Chris.
Its seems like the grand plan for Unicode is to go to UTF-32 (4 bytes), and UTF-16 (2 bytes) is already the current standard for most OS's:
http://en.wikipedia.org/wiki/ UTF-16#Use_in_major_operating_systems_and_environments
.hc
On Nov 14, 2007, at 3:18 AM, Bryan Jurish wrote:
morning all,
One (potential) problem with strings-as-arrays (as has been pointed out before) is a gross waste of space: for IEEE-754 floats, sizeof (float)=4, which is 3 bytes wasted for a single 8-bit character. We could do some devious byte-packing and pointer-casting in one or more externals to attenuate the problems, but that gets very cryptic very quickly (I actually tried this with [pdstring] (which wastes even *more* memory by using one atom per character), but decided against it in the interests of clarity and flexibility)... I think it's definitely worth a go however...
marmosets, Bryan
On 2007-11-14 08:13:03, Hans-Christoph Steiner hans@eds.org appears to have written:
Using arrays as strings is an interesting idea. I don't think non- ascii charsets should be too big a deal, they are decently supported right now, without even trying :). The Pd floats should store UTF-16 fine, which really covers basically everything. By the time UTF-32 is used much, Pd will be using 64-bit floats.
.hc
On Nov 14, 2007, at 12:43 AM, Miller Puckette wrote:
HI all,
I don't have answers to all these, but I'm sure that adding a string ytpe to Pd isn't the roght way to handle these problems. But specifically:
- spaces in symbols are a parsing/formatting problem, not a data
type problem; 2. use arrays as strings as I proposed; 3. I have to think about that one some more (!) and 4. one thing is dealing with non-ascii character sets, although there are likely to be many more problems to adress.
On Tue, Nov 13, 2007 at 11:57:06PM -0500, Chris McCormick wrote:
Hi,
I have deleted Miller's reply where he said that he's not that interested in adding a string type to Pd, but I'd like to ask him a couple of questions regarding that response, if that's ok.
- How do you propose to solve the 'spaces in file path' issue
without a string type? Or are you content with that restriction?
- How do you suggest that people deal with the symbol table
pollution issue mentioned before on this list, when they are doing operations processing lots and lots of symbol-strings in Pd? Let me know if you want more information about this issue.
- Will a [symbol2list] ever make it into Pd canonical so that
people can split long symbols on a character, like the zexy external that does this? It seems strange that you can concatenate symbols, but not split them apart again.
- Can anyone else help me with a concise summary of other
string/Pd issues I haven't thought of?
Thanks for taking the time to read and reply.
Best,
Chris.
-- Bryan Jurish "There is *always* one more bug." jurish@ling.uni-potsdam.de -Lubarsky's Law of Cybernetic Entomology
------------------------------------------------------------------------ ----
Mistrust authority - promote decentralization. - the hacker ethic
On Wed, 14 Nov 2007, Hans-Christoph Steiner wrote:
Its seems like the grand plan for Unicode is to go to UTF-32 (4 bytes), and UTF-16 (2 bytes) is already the current standard for most OS's: http://en.wikipedia.org/wiki/ UTF-16#Use_in_major_operating_systems_and_environments
Ruby2 uses UTF-8 as internal representation, and both Debian and Ubuntu default to UTF-8 as external representation. OSX's filesystem uses a subset of UTF-8. Java's .class format uses a slightly nonstandard variant of UTF-8. RFC-3986 prescribes that UTF-8 be the standard encoding for %-codes in URLs and URIs in general.
Also, with UTF-8 you don't even have to think about little-endian vs big-endian, and generally you have much less problems using UTF-8 with software that assumes that you are using some 8-bit null-terminated encoding (and there's a lot of software like that).
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada
On Wed, 14 Nov 2007, Hans-Christoph Steiner wrote:
Using arrays as strings is an interesting idea. I don't think non- ascii charsets should be too big a deal, they are decently supported right now, without even trying :). The Pd floats should store UTF-16 fine, which really covers basically everything. By the time UTF-32 is used much, Pd will be using 64-bit floats.
UTF is just an encoding over variable number of bytes. Storing as pd floats is more like UCS-4, which is a unicode encoding over a fixed number of bytes, the difference being that UCS-4 uses uint32 instead of float32, or instead of float32 plus a type tag that is always set to say "this is a float", or that plus padding because of 64-bit mode.
float32 supports all integers from 0 to 16777216, so, it includes anything that uint24 can do. I don't think you need to go beyond 18 bits, let alone 24. (UTF-8 needs extra bits per byte to say whether the character continues in the next byte; those don't count here, as we'd be using a fixed size)
Afaik, the difference between UTF-8 and UTF-16 is only that the latter tends to take somewhat less space if most of your characters are not in the ASCII range (32 to 126). this is because of the extra bits I'm talking about. In practice, all you can do with UTF-16 can already be done with UTF-8.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada
Hallo, Chris McCormick hat gesagt: // Chris McCormick wrote:
- Can anyone else help me with a concise summary of other string/Pd
issues I haven't thought of?
Not really, but if you consider symbols as a kind of immutable string, then this is not uncommon as a string-like type in programming languages. It even seems to have some advantages, as this text about Lua's strings hints at: http://mooseyard.com/Jens/2005/06/lua-and-unique-strings/ In Pd, garbage collection of unused symbols is missing, though.
But beware, I'm no language design expert at all.
Ciao
On Wed, 14 Nov 2007, Frank Barknecht wrote:
Not really, but if you consider symbols as a kind of immutable string, then this is not uncommon as a string-like type in programming languages. It even seems to have some advantages, as this text about Lua's strings hints at: http://mooseyard.com/Jens/2005/06/lua-and-unique-strings/ In Pd, garbage collection of unused symbols is missing, though.
If I ever finally implement reference-count deallocation of symbols, it's going to be unique strings anyway, because it has to; else I'd have to make a string type separate from the symbol type. I just didn't know that it would bring Pd that much closer to Lua.
Once it's done, there is some further fine-tuning that can be done, so that the hash-function (used for figuring out positions in the symbol-table) doesn't take too long to compute, but at this point it's not even worth thinking about it.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada
On Tue, 13 Nov 2007, Chris McCormick wrote:
- How do you propose to solve the 'spaces in file path' issue without a
string type? Or are you content with that restriction?
just fix atom_string().
- How do you suggest that people deal with the symbol table pollution
issue mentioned before on this list, when they are doing operations processing lots and lots of symbol-strings in Pd? Let me know if you want more information about this issue.
deallocatable symbols are possible, but they require a new API for externals. Any externals using the old API would automatically force symbols to stay allocated, because that API gives them the right to keep symbol pointers forever without even having to say that they are in use, so, pd can't safely deallocate unused symbols that have been used through the existing API.
using arrays or lists to represent strings, means that strings can't be atoms, because lists can't be used as atoms either. The problem of using lists as atoms is similar to the problem of deallocating symbols, in my view of how those lists should work, but an alternate solution would be to recursively copy all sublists of a message whenever an object wants to store a message. The latter is less efficient but easier to implement.
- Can anyone else help me with a concise summary of other string/Pd
issues I haven't thought of?
Some encodings of unicode need a distinction between number of bytes and number of characters. some other encodings of unicode use 2 bytes for every character. use of multiple encodings at once can be troublesome. use of unicode and non-unicode encodings at once can be even more troublesome. some special numbers in unicode are not characters, they are character modifiers. Tk 8.4's unicode support is not as complete as Tk 8.5's.
Storing strings as lists of floats waste a lot of memory: a character takes 1 to 3 bytes, perhaps 4, whereas an atom uses 8 bytes in 32-bit mode and 16 bytes in 64-bit mode. Arrays are more efficient, but you don't get below 4 bytes, and you can't pass them as messages.
Converting from float atom lists to C strings and back, take some time that could be reduced by using the same internal format (e.g. symbols are like that).
In addition to space problems, Pd objects often have trouble with braces, backslashes and such.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada