The current pdstring format of lists of bytes seems to be working out pretty well. I just had a thought, it would be nice to be able to write Unicode text out as UTF-8 files using [binfile]. I think currently it only accepts byte values of 0-255. What if it converted values > 255 in the UTF-8 style? Or should that be a separate [utf8file]? I suppose the advantage of [utf8file] would be to make an easy conversion to [utf16file] and [utf32file].
.hc
----------------------------------------------------------------------------
"It is convenient to imagine a power beyond us because that means we don't have to examine our own lives.", from "The Idols of Environmentalism", by Curtis White
On 2012-01-30 17:45, Hans-Christoph Steiner wrote:
The current pdstring format of lists of bytes seems to be working out pretty well. I just had a thought, it would be nice to be able to write Unicode text out as UTF-8 files using [binfile]. I think currently it only accepts byte values of 0-255. What if it converted values> 255 in the UTF-8 style? Or should that be a separate [utf8file]? I suppose the advantage of [utf8file] would be to make an easy conversion to [utf16file] and [utf32file].
[binfile] can work with any kind of file. It seems simpler to have filters that could convert between lists of bytes and lists of utf character codes, the way [slipenc] and [slipdec] will operate on any stream of bytes. Then they wouldn't be tied to [binfile], and binfile wouldn't get all bloated with obscure variations. Martin
On Jan 30, 2012, at 6:36 PM, Martin Peach wrote:
On 2012-01-30 17:45, Hans-Christoph Steiner wrote:
The current pdstring format of lists of bytes seems to be working out pretty well. I just had a thought, it would be nice to be able to write Unicode text out as UTF-8 files using [binfile]. I think currently it only accepts byte values of 0-255. What if it converted values> 255 in the UTF-8 style? Or should that be a separate [utf8file]? I suppose the advantage of [utf8file] would be to make an easy conversion to [utf16file] and [utf32file].
[binfile] can work with any kind of file. It seems simpler to have filters that could convert between lists of bytes and lists of utf character codes, the way [slipenc] and [slipdec] will operate on any stream of bytes. Then they wouldn't be tied to [binfile], and binfile wouldn't get all bloated with obscure variations. Martin
That does make more sense, so something like [bytes2utf8], etc.
.hc
----------------------------------------------------------------------------
Access to computers should be unlimited and total. - the hacker ethic
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-01-31 00:52, Hans-Christoph Steiner wrote:
That does make more sense, so something like [bytes2utf8], etc.
utf8 is always a list of bytes. if you get values >255 than it is not utf-8; do you mean unicode points?
fgmasdr IOhannes
On 2012-01-31 09:41, IOhannes m zmoelnig wrote:
On 2012-01-31 00:52, Hans-Christoph Steiner wrote:
That does make more sense, so something like [bytes2utf8], etc.
utf8 is always a list of bytes. if you get values >255 than it is not utf-8; do you mean unicode points?
Assumedly. There's already [wchars2bytes] in pdstring, which will convert wchar_t codepoints to a locale-dependent byte-string, but this is very system dependent. the function u8_toucs() from s_utf8.c should perform *exactly* a unicode-codepoint-string to utf8-byte-string conversion; the function u8_wc_toutf8() converts a single (unicode) character.
marmosets, Bryan
[apologies for multiple posts; i'm still redistributing my email accounts]
On 2012-01-31 03:41, IOhannes m zmoelnig wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-01-31 00:52, Hans-Christoph Steiner wrote:
That does make more sense, so something like [bytes2utf8], etc.
utf8 is always a list of bytes. if you get values>255 than it is not utf-8; do you mean unicode points?
The same idea could be used for unicode, converting pairs of bytes into single numbers, taking care of endianness.
Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-01-31 15:41, Martin Peach wrote:
On 2012-01-31 03:41, IOhannes m zmoelnig wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-01-31 00:52, Hans-Christoph Steiner wrote:
That does make more sense, so something like [bytes2utf8], etc.
utf8 is always a list of bytes. if you get values>255 than it is not utf-8; do you mean unicode points?
The same idea could be used for unicode, converting pairs of bytes into single numbers, taking care of endianness.
yes, something like [1] which does it the other way round (and doesn't care about endianness)
and btw, isn't utf-8 agnostic of byte-endianness? at least [2] suggests this.
mfgasdr IOhannes
[1] https://pure-data.svn.sourceforge.net/svnroot/pure-data/trunk/externals/iem/...
[2] http://en.wikipedia.org/wiki/Byte_order_mark#UTF-8
Le 2012-01-31 à 16:18:00, IOhannes m zmoelnig a écrit :
and btw, isn't utf-8 agnostic of byte-endianness? at least [2] suggests this.
UTF-8, despite the existence of something named BOM (Byte Order Mark), only has a single version.
The BOM was invented to distinguish endianness of the two UTF-16 encodings : the fully big-endian one, and the «little-endian» one (which is actually a middle-endian because beyond byte-order of 16-bit chunks, UTF is always big-endian).
However, encoding the BOM as the first character of a document or string also allows to unambiguously distinguish UTF-8, both kinds of UTF-32, UTF-7, UTF-1, and the four weirder variations of UTF. So it's useful well beyond the original idea (name) of BOM.
______________________________________________________________________ | Mathieu BOUCHARD ----- téléphone : +1.514.383.3801 ----- Montréal, QC
On Tue, Jan 31, 2012, at 09:41, IOhannes m zmoelnig wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-01-31 00:52, Hans-Christoph Steiner wrote:
That does make more sense, so something like [bytes2utf8], etc.
utf8 is always a list of bytes. if you get values >255 than it is not utf-8; do you mean unicode points?
utf16 and utf32 are also lists of bytes. It would have to be something like [utf8bytes2any], [utf16bytes2any], etc.
.hc
-
Bryan Jurish -
Hans-Christoph Steiner -
IOhannes m zmoelnig -
Martin Peach -
Mathieu Bouchard