morning all,
Would anyone object if the [any2string] semantics were changed so that only "unsigned char" values in the range (0..255) get output, rather than (as is currently the case) "signed char" values in the range (-128..127)?
For reasons to do so other than the purely aesthetic, see Roman's recent sourceforge report wrt. [any2string] (Bug #2501709, http://sourceforge.net/tracker/index.php?func=detail&aid=2501709&gro...).
speak now or forever live with the consequences, etc. etc., Bryan
On Thu, 15 Jan 2009, Bryan Jurish wrote:
Would anyone object if the [any2string] semantics were changed so that only "unsigned char" values in the range (0..255) get output, rather than (as is currently the case) "signed char" values in the range (-128..127)?
I would object, as I expect to be able to put values in the range 0 to 1114111, or at the very least the range of Unicode that people would use... 65535 is probably not enough. I'd recommend storing strings as either UCS-4 or UTF-8, but in the latter case you have variable number of bytes to take care of. Internally, I believe that UCS-4 (32-bit encoding) is full good, as Pd's lists of floats are gonna be encoded over 64-bits or 128-bits anyway (wasting nearly half or 3/4 of the bits depending on whether you have a 32-bit or 64-bit OS/mode).
What's important to me is that the Pd user does not struggle with making pd interpret UTF-8 variable-length encoding, and instead struggles with making pd work with lists of characters, which is already enough work anyway. I like that [list length] gives me the number of characters and not the number of bytes, because the latter is rarely significant.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
Mathieu Bouchard wrote:
On Thu, 15 Jan 2009, Bryan Jurish wrote:
Would anyone object if the [any2string] semantics were changed so that only "unsigned char" values in the range (0..255) get output, rather than (as is currently the case) "signed char" values in the range (-128..127)?
I would object, as I expect to be able to put values in the range 0 to
so does anybody object to use an "unsigned" type rather than a signed one?
expanding "uchar" to "uint" or whatever is no-work on the Pd-side of things.
gbmdsr IOhannes
On Thu, 15 Jan 2009, IOhannes m zmoelnig wrote:
so does anybody object to use an "unsigned" type rather than a signed one? expanding "uchar" to "uint" or whatever is no-work on the Pd-side of things.
It's not that, it's that if you have ü (u umlaut) taken from a UTF-8 file, then do you treat is as 195 188, or as 252 ? That is, is it predominantly two bytes, or predominantly one character ?
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
moin again all,
On 2009-01-15 20:37:12, Mathieu Bouchard matju@artengine.ca appears to have written:
On Thu, 15 Jan 2009, IOhannes m zmoelnig wrote:
so does anybody object to use an "unsigned" type rather than a signed one? expanding "uchar" to "uint" or whatever is no-work on the Pd-side of things.
It's not that, it's that if you have ü (u umlaut) taken from a UTF-8 file, then do you treat is as 195 188, or as 252 ? That is, is it predominantly two bytes, or predominantly one character ?
To clarify: my position is that the most fundamental representation is the raw byte string, so a UTF-8 'ü' would be represented as bytes(encode("utf8",'ü'))={195,188}. Nothing stands in the way of parsing unicode codepoint values from such a representation, to get unicode_chars("utf8",{195,188})={252}.
OTOH, if the file were known to be encoded in latin-1, we'd have bytes(encode("latin1",'ü'))={252}. Without knowing the encoding of the source, there's no reliable way to determine which unicode codepoints its raw bytes are representing (or even if the powers that be have seen fit to define such codepoints), so I would argue against making unicode the *only* available internal representation.
marmosets, Bryan
moin Mathieu, moin all,
On 2009-01-15 16:33:03, Mathieu Bouchard matju@artengine.ca appears to have written:
On Thu, 15 Jan 2009, Bryan Jurish wrote:
Would anyone object if the [any2string] semantics were changed so that only "unsigned char" values in the range (0..255) get output, rather than (as is currently the case) "signed char" values in the range (-128..127)?
What's important to me is that the Pd user does not struggle with making pd interpret UTF-8 variable-length encoding, and instead struggles with making pd work with lists of characters, which is already enough work anyway.
Agreed (in principle at least)... At the risk of repeating myself, I wrote [any2string] and [string2any] as quick ugly hacks to get some sort of rudimentary string handling in pd. Roman mentioned a few other externals (e.g. [comport]) which expect unsigned raw byte values, which I think is sufficient reason to change the (byte-oriented) conventions of [any2string].
Unicode might be more immediately intuitive to most users, but when it comes down to it, byte-strings are IMHO the more basic representation (a char* is still a char*, even in this post-unicode world). Some of us even still use non-unicode encodings by default. A good string handling mechanism should have a good general default representation (e.g. as UTF-${MachineWordBits}), but should likewise allow access to "raw" byte strings, and be able to accommodate various encodings. Not that I'm really hankering to write any of that, mind you ;-)
Perhaps a better name for the external as I think of it would be [any2bytes]. I'm perfectly willing to cede the "string" name to something better (Martin's string patch comes to mind), but that's just a labelling issue (and since variable names are arbitrary, and externals are in some sense variables, external names must therefore also be arbitrary ;-)
I like that [list length] gives me the number of characters and not the number of bytes, because the latter is rarely significant.
... except if you're building rsp. reading a persistent index for a large file, in which case tell() & seek() are likely to be a wee bit faster than parsing and counting variable-length-encoded characters ...
marmosets, Bryan
On Thu, 15 Jan 2009, Bryan Jurish wrote:
Unicode might be more immediately intuitive to most users, but when it comes down to it, byte-strings are IMHO the more basic representation (a char* is still a char*, even in this post-unicode world).
What happened is that people switched to UTF-8 instead of some fixed-size encoding because many apps that assume that a character is a byte will work anyway. Just don't ask those apps to say how many characters there are in a string though. You have to pretend that all the "special" characters are pairs of characters instead (when they are not triplets).
A good string handling mechanism should have a good general default representation (e.g. as UTF-${MachineWordBits}), but should likewise allow access to "raw" byte strings, and be able to accommodate various encodings. Not that I'm really hankering to write any of that, mind you ;-) Perhaps a better name for the external as I think of it would be [any2bytes]. I'm perfectly willing to cede the "string" name to something better (Martin's string patch comes to mind),
I gather that it'll take a long time before Pd gets unicode support...
... except if you're building rsp. reading a persistent index for a large file, in which case tell() & seek() are likely to be a wee bit faster than parsing and counting variable-length-encoded characters ...
right.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
Unicode would be great, if someone wants to implement that in Pd, I
would fully support that.
As for the simple question of unsigned versus signed for any2string, I
have never heard of using negative values for chars, so it makes
perfect sense to me to use 0-255. That'll will give at least the full
latin charset.
.hc
On Jan 15, 2009, at 12:45 PM, Mathieu Bouchard wrote:
On Thu, 15 Jan 2009, Bryan Jurish wrote:
Unicode might be more immediately intuitive to most users, but when
it comes down to it, byte-strings are IMHO the more basic
representation (a char* is still a char*, even in this post-unicode world).What happened is that people switched to UTF-8 instead of some fixed- size encoding because many apps that assume that a character is a
byte will work anyway. Just don't ask those apps to say how many
characters there are in a string though. You have to pretend that
all the "special" characters are pairs of characters instead (when
they are not triplets).A good string handling mechanism should have a good general default
representation (e.g. as UTF-${MachineWordBits}), but should
likewise allow access to "raw" byte strings, and be able to
accommodate various encodings. Not that I'm really hankering to
write any of that, mind you ;-) Perhaps a better name for the
external as I think of it would be [any2bytes]. I'm perfectly
willing to cede the "string" name to something better (Martin's
string patch comes to mind),I gather that it'll take a long time before Pd gets unicode support...
... except if you're building rsp. reading a persistent index for a large file, in which case tell() & seek() are likely to be a wee bit faster than parsing and counting variable-length-encoded
characters ...right.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal,
Québec_______________________________________________ Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
All information should be free. - the hacker ethic
morning all,
On 2009-01-15 22:51:14, Hans-Christoph Steiner hans@eds.org appears to have written:
As for the simple question of unsigned versus signed for any2string, I have never heard of using negative values for chars, so it makes perfect sense to me to use 0-255. That'll will give at least the full latin charset.
OK. I guess I'll make unsigned values the default for [any2string] then, postponing the issue of name change to [any2bytes] rsp. [bytes2any] to a hypothetical future in which the "string" suffix implies unicode or other non-byte-oriented representation.
marmosets, Bryan
On Sat, 17 Jan 2009, Bryan Jurish wrote:
OK. I guess I'll make unsigned values the default for [any2string] then, postponing the issue of name change to [any2bytes] rsp. [bytes2any] to a hypothetical future in which the "string" suffix implies unicode or other non-byte-oriented representation.
Make an alias to [any2bytes] anyway, and encourage people to make the distinction right away, so that [any2string] in the future would take care of encodings while [any2bytes] would not.
Or else, [any2string] could take an argument for the charset, maybe.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
Mathieu Bouchard wrote:
On Sat, 17 Jan 2009, Bryan Jurish wrote:
OK. I guess I'll make unsigned values the default for [any2string] then, postponing the issue of name change to [any2bytes] rsp. [bytes2any] to a hypothetical future in which the "string" suffix implies unicode or other non-byte-oriented representation.
Make an alias to [any2bytes] anyway, and encourage people to make the distinction right away, so that [any2string] in the future would take care of encodings while [any2bytes] would not.
Or else, [any2string] could take an argument for the charset, maybe.
Am I missing something here? How does one make a symbol or a message or anything at all in pd using for example chinese or arabic characters? I thought it was ASCII all the way, and even then there are reserved characters that get snagged by tcl.
The term 'string' seems to have two separate meanings in this thread. One designates a string of text, for presentation. The other designates a string of bytes, for transmission. I think [any2string] is intended for purposes of serialization, to send data through a byte-oriented channel such as a serial port, ethernet, or file.
Maybe for text an [any2symbol] is what is needed. It would convert text in any encoding to a valid pd symbol. The symbol could then be passed through [any2string] to get the raw bytes.
Martin
Martin Peach wrote:
a string of bytes, for transmission. I think [any2string] is intended for purposes of serialization, to send data through a byte-oriented channel such as a serial port, ethernet, or file.
i always thought that bryan's "pdstrings" was intended for purposes of linguistic processing (unlike a lot of your objects that are indeed targeted at transmission).
nfga,sdr IOhannes
On Sun, 2009-01-18 at 13:46 +0100, IOhannes m zmoelnig wrote:
Martin Peach wrote:
a string of bytes, for transmission. I think [any2string] is intended for purposes of serialization, to send data through a byte-oriented channel such as a serial port, ethernet, or file.
i always thought that bryan's "pdstrings" was intended for purposes of linguistic processing (unlike a lot of your objects that are indeed targeted at transmission).
without having thought of what pdstring could be targeted to, i was always looking for something targeted for transmission, i.e. to be used together with [comport] or [tcp[server|client|send|receive]]. if pdstring is not recommended, what is recommended then?
roman
___________________________________________________________ Der frühe Vogel fängt den Wurm. Hier gelangen Sie zum neuen Yahoo! Mail: http://mail.yahoo.de
moin all,
On 2009-01-18 19:41:06, Roman Haefeli reduzierer@yahoo.de appears to have written:
On Sun, 2009-01-18 at 13:46 +0100, IOhannes m zmoelnig wrote:
Martin Peach wrote: i always thought that bryan's "pdstrings" was intended for purposes of linguistic processing (unlike a lot of your objects that are indeed targeted at transmission).
without having thought of what pdstring could be targeted to, i was always looking for something targeted for transmission, i.e. to be used together with [comport] or [tcp[server|client|send|receive]]. if pdstring is not recommended, what is recommended then?
well, without wanting to be trite, I have to say that think that "data transmission" and "linguistic processing" are pretty much synonymous. In fact, the original reason for writing [pdstring] was to enable me to transmit (pseudo-)natural-language data represented as lists of pd atoms as a stream of byte values over OSC (using zexy's [drip]), so I guess the "target use" is "transmitting (processed) character data" ;-)
That said, I think the [pdstring] representation is just a way of fudging a string datatype into pd -- use it for whatever seems useful to you. The "string" part may (as Mathieu suggested) change to unicode in the future, but I've added aliases [any2bytes] and [bytes2any] for now.
marmosets, Bryan
Bryan Jurish wrote:
moin all,
well, without wanting to be trite, I have to say that think that "data transmission" and "linguistic processing" are pretty much synonymous.
Pretty much but linguistic processing is happening at a higher level than data transmission, and the 'character' used in language may be represented by more than one data transmission 'character'. With ASCII and its 8-bit relatives it's almost the same because the two kinds of character are the same, but unicode uses more than one data character per linguistic character. That's why I think there needs to be a distinction between the two types of 'string', and maybe two levels of objects to deal with them, like [unicode2byte]. Martin
morning again all,
On 2009-01-19 15:19:04, Martin Peach martin.peach@sympatico.ca appears to have written:
Bryan Jurish wrote:
well, without wanting to be trite, I have to say that think that "data transmission" and "linguistic processing" are pretty much synonymous.
Pretty much but linguistic processing is happening at a higher level than data transmission, and the 'character' used in language may be represented by more than one data transmission 'character'.
Well, speaking as a linguist, I guess I have to say that I don't think meaningful "linguistic processing" can really happen at the character level (even though that's almost invariably the starting point for NLP programs), but that's just me getting pedantically OT. Sorry.
Of course you're right, and "characters" are intended to represent linguistically salient units ("graphemes").
With ASCII and its 8-bit relatives it's almost the same because the two kinds of character are the same, but unicode uses more than one data character per linguistic character.
Yup (well, sometimes it does... it's the curse of those darned variable-length encodings again...)
That's why I think there needs to be a distinction between the two types of 'string', and maybe two levels of objects to deal with them, like [unicode2byte].
I fully agree: we should distinguish between "byte strings" and "(unicode) character strings". As for converting between character- and byte-strings, there are a whole slew of encoding pitfalls to watch out for. Converting between (say) a UCS-4 unicode character string and a UTF-8 byte string is easy, but things get uglier if we want an old-style 8-bit encoding (other than latin-1) for the byte string...
marmosets, Bryan
On Jan 17, 2009, at 1:58 PM, Martin Peach wrote:
Mathieu Bouchard wrote:
On Sat, 17 Jan 2009, Bryan Jurish wrote:
OK. I guess I'll make unsigned values the default for [any2string] then, postponing the issue of name change to [any2bytes] rsp. [bytes2any] to a hypothetical future in which the "string" suffix implies unicode or other non-byte-oriented representation.
Make an alias to [any2bytes] anyway, and encourage people to make the distinction right away, so that [any2string] in the future would take care of encodings while [any2bytes] would not.
Or else, [any2string] could take an argument for the charset, maybe.
Am I missing something here? How does one make a symbol or a message
or anything at all in pd using for example chinese or arabic characters? I thought it was ASCII all the way, and even then there are reserved characters that get snagged by tcl.The term 'string' seems to have two separate meanings in this thread. One designates a string of text, for presentation. The other
designates a string of bytes, for transmission. I think [any2string] is intended for purposes of serialization, to send data through a byte-oriented channel such as a serial port, ethernet, or file.Maybe for text an [any2symbol] is what is needed. It would convert
text in any encoding to a valid pd symbol. The symbol could then be passed through [any2string] to get the raw bytes.Martin
I think Pd just filters { } \ and all other characters go thru... kind
of. Latin1 definitely works, and I've seen some other characters work
in Pd patches. This is one goal of the current pd-devel effort:
removing arbitrary ASCII restrictions. Tcl/Tk is unicode, so I don't
think it would be too hard to make Pd fully Unicode, if someone wanted
to take that on. Having the GUI pure Tcl should make that easier.
.hc
'You people have such restrictive dress for women,’ she said, hobbling
away in three inch heels and panty hose to finish out another pink-
collar temp pool day. - “Hijab Scene #2", by Mohja Kahf
On Jan 17, 2009, at 5:37 AM, Bryan Jurish wrote:
morning all,
On 2009-01-15 22:51:14, Hans-Christoph Steiner hans@eds.org
appears to have written:As for the simple question of unsigned versus signed for
any2string, I have never heard of using negative values for chars, so it makes
perfect sense to me to use 0-255. That'll will give at least the full latin charset.OK. I guess I'll make unsigned values the default for [any2string] then, postponing the issue of name change to [any2bytes] rsp. [bytes2any] to a hypothetical future in which the "string" suffix implies unicode or other non-byte-oriented representation.
Well, C considers a collection of 0-127 values a string. I think most
programming languages would call an collection of 8-bit ASCII bytes a
string. I think "string" makes more sense than "bytes" here.
.hc
marmosets, Bryan
-- Bryan Jurish "There is *always* one more
bug." jurish@ling.uni-potsdam.de -Lubarsky's Law of Cybernetic
Entomology
Man has survived hitherto because he was too ignorant to know how to
realize his wishes. Now that he can realize them, he must either
change them, or perish. -William Carlos Williams
moin Mathieu, moin all,
On 2009-01-15 20:45:13, Mathieu Bouchard matju@artengine.ca appears to have written:
On Thu, 15 Jan 2009, Bryan Jurish wrote:
byte-strings are IMHO the more basic representation (a char* is still a char*, even in this post-unicode world).
What happened is that people switched to UTF-8 instead of some fixed-size encoding because many apps that assume that a character is a byte will work anyway.
UTF-8 also does a pretty good job of compactly representing latin character sets for natural language data, where non-ASCII characters tend to be relatively infrequent anyways. UTF-16 and UTF-32 are pretty wasteful in these cases. (Of course, I'm biting my own tail with this point, since the [pdstring] representation is even more wasteful than UTF-32 ;-)
Just don't ask those apps to say how many characters there are in a string though. You have to pretend that all the "special" characters are pairs of characters instead (when they are not triplets).
Indeed. Ugly but true.
I gather that it'll take a long time before Pd gets unicode support...
I suspect you're right.
... except if you're building rsp. reading a persistent index for a large file, in which case tell() & seek() are likely to be a wee bit faster than parsing and counting variable-length-encoded characters ...
right.
... or calling malloc(), or doing pretty much any other low-level fiddly stuff ...
marmosets, Bryan
On Fri, 16 Jan 2009, Bryan Jurish wrote:
UTF-8 also does a pretty good job of compactly representing latin character sets for natural language data, where non-ASCII characters tend to be relatively infrequent anyways. UTF-16 and UTF-32 are pretty wasteful in these cases. (Of course, I'm biting my own tail with this point, since the [pdstring] representation is even more wasteful than UTF-32 ;-)
Well, RAM is in discount at a very big mailorder store, where you can get 4096 megs as two sticks of DDR2-800 memory for 29,99 CAD, which is 18,24 EUR.
I don't think that the goal is to be compact, nor that you really have much choice here. The goal is so that Pd users can mess with string characters the way they want, in a way that is fairly easy to use (well at least, that's the goal I can infer when I look at the idea of using lists as strings!). Then if you decide not to depend on any other Pd library and try to leverage existing Pd 0.39+ the most you can, then you have to use Pd's lists, and then it's 64 or 128 bits per char.
And then, in theory, Pd could adopt any internal rep, as long as file I/O and socket I/O is done the way it needs to be done.
... except if you're building rsp. reading a persistent index for a large file, in which case tell() & seek() are likely to be a wee bit faster than parsing and counting variable-length-encoded characters ...
right.
... or calling malloc(), or doing pretty much any other low-level fiddly stuff ...
It doesn't matter much, as Pd patches wouldn't be doing malloc(). Furthermore, I expect that you have or you would have a function for converting a list to a C string in the proper encoding, so that externs that want to use your strings don't have to do for(i=0;...) a[i]=b[i] all of the time, but also because it's a good opportunity for introducing optional encoding conversion.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
moin moin,
On 2009-01-16 15:56:22, Mathieu Bouchard matju@artengine.ca appears to have written:
you have to use Pd's lists, and then it's 64 or 128 bits per char.
And then, in theory, Pd could adopt any internal rep, as long as file I/O and socket I/O is done the way it needs to be done.
... which (if I understand correctly) pushes the whole encoding mess onto the I/O layer, which I believe (based on many many past headaches trying to get the encoding support of the perlIO layer to work transparently on in-memory strings) is The Wrong Way To Do It (TM). Precisely the I/O layer is "low-level" in my sense, which means it ought to be bytes only. Encoding-dependent "character" units are higher-level and ought to be independent of that layer.
... except if you're building rsp. reading a persistent index for a large file, in which case tell() & seek() are likely to be a wee bit faster than parsing and counting variable-length-encoded characters ...
right.
... or calling malloc(), or doing pretty much any other low-level fiddly stuff ...
It doesn't matter much, as Pd patches wouldn't be doing malloc(). Furthermore, I expect that you have or you would have a function for converting a list to a C string in the proper encoding, so that externs that want to use your strings don't have to do for(i=0;...) a[i]=b[i] all of the time, but also because it's a good opportunity for introducing optional encoding conversion.
Leveraging vanilla pd means that I can't (easily) export any functions, since each external is supposed to be self-contained. Of course, it's easy to write such functions and offer them as "copy-in" replacements, or define function-body macros, etc. etc. ... to date, there have been no requests for such an API, and potential users have to write their own for-loops...
marmosets, Bryan
-
Bryan Jurish -
Hans-Christoph Steiner -
IOhannes m zmoelnig -
Martin Peach -
Mathieu Bouchard -
Roman Haefeli