moin all,

@hans: thanks for the vote of confidence :-)

@João: ... for converting strings (lists of bytes, represented as pd floats) to floats (as in pd floats, a la C strtof() & friends) you can do some sick bad ugly and wrong things using [bytes2any] in conjunction with [locale] (e.g. by dynamically re-setting LC_NUMERIC) ... but maybe that wasn't what you meant...

marmosets, Bryan

On 2011-01-26 21:06:35, Hans-Christoph Steiner hans@at.or.at appears to have written:

Pd < 0.43 is probably something like ISO-8859-1 or perhaps UTF-8, Pd 0.43 is UTF-8. UTF-8 or ISO-8859-1 will both be fully ANSI if only the standard ASCII chars are used, i.e. no ü, ã, é, etc.

For converting, I like moocow/any2bytes and moocow/bytes2any.

.hc

On Jan 26, 2011, at 5:17 AM, João Pais wrote:

...

Hi,

I was curious to know what is the standard encoding of pd files in each operating system. According to Notepad+, in XP is ANSI.

Or a followup question, which format is better for pd to read a string of characters and convert it to floats? ANSI seems the be most reliable here where it comes to interpreting the data, although utf8 without BOM renders them in the symbol atom more accurately.

The reason I'm asking is because I'm working with converting ascii characters to their float value.

If anyone wants, I can send a small test patch to try out.

João

-- Friedenstr. 58 10249 Berlin (Deutschland) Tel +49 30 42020091 | Mob +49 162 6843570 Studio +49 30 69509190 jmmmpais@googlemail.com | skype: jmmmpjmmmp

---

Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

---

If you are not part of the solution, you are part of the problem.

---

Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

Pd < 0.43 is probably something like ISO-8859-1 or perhaps UTF-8, Pd
0.43 is UTF-8. UTF-8 or ISO-8859-1 will both be fully ANSI if only the
standard ASCII chars are used, i.e. no Ì, ã, é, etc.

hmm, for my case I'm needing the 256 values in the chart. I'm noticing
that there are differences between mac, xp and ubuntu, but can't tell
exactly where/what is the cause for it.

For converting, I like moocow/any2bytes and moocow/bytes2any.

I think I had a look at it as well. do you have any comparative reason for
that one instead of the other? or it was just the first one to get to you?

moin all,

fwiw, I'll add my vote in favor of getting Martin's [str] / blob patch into pd vanilla. iirc, Miller has indicated in the past that he feels this sort of thing should be done using arrays. There are a couple of proof-of-concept objects (not compiled by default) in pdstring (moocow/pdstring for pd-extended users) that use arrays, but I wasn't satisfied with this approach for two reasons:

1. If you treat the array internally as a string of bytes (e.g. char*),

then: (A) you must constantly cast and re-cast the data (B) you must scale all size attributes (e.g. for re-allocation) by 1.0/sizeof(t_float), so to get an accurate byte length that is not a multiple of sizeof(t_float), you need to actually store that length additionally somewhere else (C) saving array data with a patch and re-loading can cause data loss (float truncation may mess up raw byte values) (D) it's not really portable (byte order problems with load/save)

2. If otoh you let the array remain a t_float* and just assign the

floats byte ((unsigned) char) or even wide character (wchar) values, then: (A) you potentially waste a lot of memory (strlen(str)*(sizeof(float)-1) bytes) (B) I/O: if you read a string as a (char*) -- e.g. from a file, socket, external library, etc. -- or if you need a pd-array-string as a (char*) -- e.g. for an external API call -- then you have to explicitly convert it, which means allocating some memory (maybe defining a static local buffer to avoid malloc() calls), and iterating over the string (rsp. over the array), which is O(N) time and space, and is annoying for long strings and/or frequent calls (C) if you really want to store your string data in an array, you can use [str] or [pdstring] together with e.g. [tabdump] and [tabset] from zexy, which just makes the conversion overhead explicit.

I think there are workarounds for both techniques, but not without patching the pd core code, and if we're going to patch the core code, we might as well take a patch that does the job "right" (i.e. Martin's)...

just my €0.02 ...

marmosets, Bryan

On 2011-01-28 09:59:39, Roman Haefeli reduzent@gmail.com appears to have written:

On Wed, 2011-01-26 at 22:54 +0100, João Pais wrote:

...

...

For converting, I like moocow/any2bytes and moocow/bytes2any.

I think I had a look at it as well. do you have any comparative reason for
that one instead of the other? or it was just the first one to get to you?

One important thing to know is that [mrpeach/str] only works with Pd-extended, but not with Pd-vanilla as it requires some modifications to m_pd.h.

Btw.... @Martin: Do you think it's time again to to try to get the blob support into Pd-vanilla? I think that [str] would be utterly useful also in vanilla and it didn't seem to have caused any problem in Pd-extended, did it?

Roman

---

Pd-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list

moin Mathieu, moin all,

On 2011-01-29 17:12:02, Mathieu Bouchard matju@artengine.ca appears to have written:

On Fri, 28 Jan 2011, Bryan Jurish wrote:

...

iirc, Miller has indicated in the past that he feels this sort of thing should be done using arrays.

But a feeling is but a feeling. Now, how about a justification ? But that's not the sort of thing one gets from Miller often.

...

(B) you must scale all size attributes (e.g. for re-allocation) by 1.0/sizeof(t_float), so to get an accurate byte length that is not a multiple of sizeof(t_float), you need to actually store that length additionally somewhere else

sizeof(t_float) is always a power of two, isn't it ? I haven't heard of anyone using 80-bit or 96-bit floats as t_float or t_sample.

thus a size stored as float will be accurate up to 16777216.

This is regardless of whether you store size*4, size, or size/4 : floats are quite scale-independent, but are perfectly so when the scalings are powers of two (provided you don't overflow by scaling by pow(2,128) or so)

I think you could read a bit about the IEEE-754 standard : http://en.wikipedia.org/wiki/IEEE_754

But especially some kind of short, direct tutorial that will make it obvious what won't be rounded and what will be : http://kipirvine.com/asm/workbook/floating_tut.htm

Yup, all freely stipulated. My issue was not so much with the use of floats qua floats to store size data, rather the necessity of storing size data *in addition to* the size reported by the array itself. In this scenario, we're blatantly re-casting the array's (t_float*) into a (char*) and reading/writing raw bytes. But maybe we don't want C-style NUL-terminated strings, but rather perl-ish (or Berkeley DB-ish) strings which admit embedded NULs and store their length in an additional dedicated attribute (usually an unsigned int, but sure, we could use a float if we wanted). The problem is that if we (ab)use the existing garray API (garray_getfloatarray(), garray_npoints(), garray_resize()) to do this, then the sizes reported for the array may be longer than the size of the string. My system uses 32-bit floats, and say I want a string "foo" (without terminating NUL). Well, "foo" takes up less than the space of 1 float (3 bytes < 4 bytes), but garray_npoints() for a float array of size 1 is going to give me 1, and 1*sizeof(float) = 4 > 3, so if I want to implement strings this way, I've got to fiddle around with some additional convention for storing their actual length.

It looks as if the whole garray stuff is defined abstractly enough to handle more than just "plain" float arrays, but I haven't dug deep enough to figure out what exactly those (t_template*)s are all about or how I might be able to (ab)use them...

...

(C) saving array data with a patch and re-loading can cause data loss (float truncation may mess up raw byte values)

for integers, all values from -1000000 to 1000000 will be correctly saved (those two bounds will be encoded as -1e+6 and 1e+6, and all the rest will look like plain integers).

Yes. See above re:

char *s; garray_getfloatarray(a,size,(float **)&s);

...

(D) it's not really portable (byte order problems with load/save)

byte order problems won't happen with floats saved as text. they will happen with floats saved as binary. they will also happen with UCS-2 text saved as two floats per code point (no matter how you save the floats), but if you use UTF-8 instead, or if you use one-float-per-codepoint, that aspect will be safe.

No. See above. Messing about with typecasts is very implementation-dependent, and afaik IEEE-754 doesn't define how its components are to be implemented, only the formal criteria an implementation must satisfy.

...

2. If otoh you let the array remain a t_float* and just assign the

floats byte ((unsigned) char) or even wide character (wchar) values, then: (A) you potentially waste a lot of memory (strlen(str)*(sizeof(float)-1) bytes)

In 2011, wasting a lot of RAM is not a problem. Wasting too much RAM can be a problem, and that's very relative, as quite often, the solution is to wait until RAM is less expensive. I like the idea of not wasting any RAM, but I recognise that this is because I got used to think about ways to reduce waste, not because it's always good to worry about it.

Text is usually a lot smaller than video. It's not uncommon for me to store a buffer of 64 frames of video in colour. In 640x480, that's over 55 megs, and that's tiny compared to the total amount of RAM the computer has. How often do you need that much text at once in RAM ?

Stipulated for most purposes. Taking ratts as an example, the CMU dictionary is only 3.5M, the beep dictionary is 7.6M. The non-free German dictionary BOMP is still only 9.1M. I agree that none of this is going to "make the cabbage any fatter", as a saying here goes. In other work, I need much more data. The morphology transducer I use is 153M stored offline. (and more at runtime). A simple word trigram model bootstrapped from a decent sized corpus can run into the hundreds of MB (the little one I have on hand in only 26MB) ... have a look at the google n-grams for an idea of where that leads when you add lots more data. Basically, the moral is: mixing Zipf's law and polynomial growth with respect to vocabulary size (e.g. n-gram models) can get you in a deep hole very very quickly. fwiw, the raw text of the whole corpus I work with these days runs about 1G. A single file with all intermediate data can easily run over 400MB. I really wouldn't want to go to N*4 there...

...

(C) if you really want to store your string data in an array, you can use [str] or [pdstring] together with e.g. [tabdump] and [tabset] from zexy, which just makes the conversion overhead explicit.

GridFlow's grids support the byte format (unsigned char). This is one of the six allowed grid formats, and perhaps the 2nd most used (after signed int).

But GridFlow isn't vanilla either.

...

I think there are workarounds for both techniques, but not without patching the pd core code, and if we're going to patch the core code, we might as well take a patch that does the job "right" (i.e. Martin's)...

If all of this can work as externals without hairy workarounds, then you don't need to be obsessing about patching pd's core code, and that's a good thing, especially if you aim to be patching vanilla's.

I think it probably can, but it's likely to amount to dependence on more than offered by just pd-vanilla (e.g. GridFlow, Martin's patch, etc.). Also, I have yet to come up with a satisfying way to make an easily extensible string-handling library for pd -- anyone wanting to handle strings in other externals should ideally also have access to some common string handling API, without having to resort to the pd-patch level, and the byte-values-in-floats approach just doesn't handle that well. I suppose I could make an array-like external specifically for handling string buffers, but that's essentially what Martin's patch does, so why re-invent the wheel?

marmosets, Bryan

moin again,

On 2011-01-30 17:07:44, Mathieu Bouchard matju@artengine.ca appears to have written:

On Sat, 29 Jan 2011, Bryan Jurish wrote:

...

In this scenario, we're blatantly re-casting the array's (t_float*) into a (char*) and reading/writing raw bytes.

Ok, I thought you were going to write one codepoint per float and not do any reinterpret_cast.

I'd advise against relying on reinterpret_cast hacks like this, and instead, add support for other number types in the t_array struct and supporting functions. In that case it'd become significantly closer to struct Grid, but without the support for multiple dimensions of indices.

I agree that sounds like the best way within pd-vanilla as it appears currently to be constructed, but I'm not at all sure about how to go about it. And up until ca. 11:14 AM this morning, I had semi-major post-catholic guilt attacks every time I even thought about doing something involving a computer keyboard that didn't directly involve either my dissertation or my day job. Happily, the former is out of my hands now (fanfare, please :-)

This change could also prevent wasting half of the t_array memory when storing floats on 64-bit computers, which is currently a good source of ridicule.

Agreed.

...

Yes. See above re: char *s; garray_getfloatarray(a,size,(float **)&s);

This is "deprecated" since 0.41, but really, this is a function call that never ever worked properly in 64-bit mode. You need to use garray_getfloatwords instead, which returns a t_word.

Yes, I did see that the underlying data was stored as a t_word; I only briefly re-grepped the sources when composing my mail... I haven't really dedicated a whole lot of time to this attempt, as I think you can probably tell (bad hacker, no biscuit...)

...

No. See above. Messing about with typecasts is very implementation-dependent, and afaik IEEE-754 doesn't define how its components are to be implemented, only the formal criteria an implementation must satisfy.

If you (or pd) never actually read the contents as float values in your use of reinterpret_cast to store, it doesn't matter, as you're doing nothing with float* that may depend on the difference between, say, float* and char*.

True. But while I can guarantee this for "string-like" operations, I can't seem to finagle it for pd, which insists on treating arrays (at least those defined at the patch level) as t_float[]s (looks like the culprit here is garray_save() calling binbuf_addv() to buffer the array data, which assumedly gets dumped to the file at some point via binbuf_gettext(), which calls atom_string() which ends up dumping the float with the sprintf() "%g" format, i.e. truncating... but we all knew that already, right?

But you're not supposed to be using float* anymore, just t_word*, if you want to continue with the reinterpret_cast hack.

reinterpret_cast<> doesn't exist.

there's only (char*)expr (cf. Kernighan & Ritchie, 1988)

:-P

sorry for being pedantic; you are of course correct that reinterpret_cast<> is the C++ equivalent for what I'm doing; I just think it's too much to type, which is yet another thing I dislike about C++ ...

anyhoo, ok: I can (ab)use typecasts using (t_word*) instead of (t_float*). My original gripes (1C) and (1D) still hold: this breaks on save/load of patches, if "string" data is to be saved with its array. I can of course say "not my problem" and leave it at that, but there are still points (1A) and (1B). (1A) is really just a convenience issue (typecasting) -- if that was the only issue, I'd already have included the code in [pdstring]. (1B) is the string-length issue, and is much hairier. We can't get string length and array length always to jive, and if I add an extra object to store string length (and maybe other properties), then why don't I just dump the string data as a (char*) into that object, and leave g_array out of it entirely?

Martin's patch does just this: he adds a (length,data) pair struct t_string and a t_string* to the t_word struct. I suppose I could divorce the underlying struct from t_word, wrap it into a new object, bind a symbol, etc etc.... I still think the idea of using arrays for strings is intriguing, not least for the sheer amount of abuse potential arising from combining text bytes and audio signals in the same arrays... so no, I guess I don't want to drop the g_array idea entirely (apologies for answering my own question)...

...

deep hole very very quickly. fwiw, the raw text of the whole corpus I work with these days runs about 1G. A single file with all intermediate data can easily run over 400MB. I really wouldn't want to go to N*4 there...

4096 MB of DDR3 RAM is currently 37,99 $, going downhill. So, even with N*8, it doesn't look like the end of the word. (l)

grr.... yes yes, point taken, but I find it horribly nasty to waste more than half of the memory allocated (ok, the strings-as-lists-of-floats waste even more, but that's explicit and open about its hackery; putting byte values into floats under the hood and calling the result "string" would be cunning, devious, and underhanded hackery... or something like that)

Now we're on to method (2). The show-stopper for me here is argument (2B): external APIs. Under this method, every time I want my pd "string" as a C string, I have to explicitly convert it, and vice versa, which takes additional buffers, possibly (re-)allocations, and O(N) time. This is all likely to happen only at the control level, so maybe that's not system critical either. Being able to easily incorporate external string-processing APIs (e.g. the C library string handling routines) is a pretty big desideratum for a string handling mechanism, im(ns)ho, so I'd have to build in some shared conversion routines, buffer structures, etc. ... but also to export them beyond the confines of a single external, which I haven't tested at all yet, and am not even sure if is realistic to think about, except maybe as a static library or code base... so the whole method (2) begins to look pretty baroque as soon as it passes out of "proof-of-concept" and towards "useable API".

But I'd still advocate adding multiple number type support to arrays.

Sounds like the best approach, agreed.

...

...

GridFlow's grids support the byte format (unsigned char). This is one of the six allowed grid formats, and perhaps the 2nd most used (after signed int).

But GridFlow isn't vanilla either.

How many solutions do you want to reject ?

n-1, for some natural number n. sorry, can't be more specific yet.

Honestly, if I had a pressing need for handling large-ish amounts of text data in pd, I would probably look to GridFlow. As it is, I usually wind up trying to get all my string processing done outside of pd, and passing the data back and forth via OSC or (brace yourself) the filesystem, where the "strings" wind up as symbols, and put a good deal of stress on pd's symbol table, but hey... it explodes only very rarely...

...

I think it probably can, but it's likely to amount to dependence on more than offered by just pd-vanilla (e.g. GridFlow, Martin's patch, etc.).

Duh. Now do you really think you're saving yourself so much work by not installing easy-to-install software ? On Ubuntu and OSX, you can install both Martin's blobs and GridFlow with not much more than two clicks.

I have installed Martin's blobs. It involved only a single patch to the pd core and a re-compilation. No big deal. Last time I tried to install GridFlow (this was years ago), I was bitten by many (potential) dependencies and an old system, and gave up. I should give it a whirl again; many of its features I've heard you mention on the list at one time or another sound very useful indeed.

I am not really concerned about the number of clicks it takes to install software (I'm happy with (./configure; make; make install) myself); what I am concerned about is the *portability* of my own software. If I were to work on yet another string handler for pd, I'd like to make sure that it's got as wide a potential user base as possible. Not everyone uses pd-extended.

But I was mentioning GridFlow just to tell you what's in there. From there, not only you can decide to use GridFlow, but if you decide to instead modify Pd, you can look at how GridFlow does it : isn't that interesting ?

It is, although I usually try to avoid mucking about in other people's code.

...

I suppose I could make an array-like external specifically for handling string buffers, but that's essentially what Martin's patch does, so why re-invent the wheel?

I think that how solutions can differ in the most notable way, is about where the string is actually stored, how you can keep one, how long you can keep it (when does it get deleted) and can you make a list of strings without having one [str] object per element. Do you agree ?

I'd say those are good differentia, yes. From where I'm standing, I'd put memory footprint and compatibility with existing 3rd-party APIs (e.g. conversion to/from (char*)) at the top the list, and Martin's strings fulfill those criteria admirably. Persistence is worth thinking about, but if you can get to/from (char*), it's pretty easy to roll your own persistence code if need be.

The list-of-strings issue you brought up is very interesting indeed; I'm almost tempted to push that into a general discussion of nested data structures, but I think we've already drawn this thread far enough OT ;-)

marmosets, Bryan

On Wed, 26 Jan 2011, Hans-Christoph Steiner wrote:

Pd < 0.43 is probably something like ISO-8859-1 or perhaps UTF-8

Pd Extended 0.42 and 0.41 are system-dependent on this : ISO_8859-1 on OSX, ISO_8859-1 in Linux's pd "server", and UTF-8 in Linux's pd "client". This explains why accents in OSX patches make text (comments, objectboxes, etc) disappear on Linux, and why trying to type accents in Linux makes text disappear.

If you load GridFlow in those versions of Pd, the gridflow_unicorn module modifies Pd so that accents are recorded as UTF-8 in the patches, which makes them typable on Linux, and makes OSX's text portable to Linux. However, in both cases, the other UTF-8 problems remain (cursor in wrong place, problems with deleting and inserting special chars).

UTF-8 or ISO-8859-1 will both be fully ANSI if only the standard ASCII chars are used, i.e. no ü, ã, é, etc.

What does «ANSI» mean here ?? (I suspect that it's not what you mean to write)

| Mathieu Bouchard ---- tél: +1.514.383.3801 ---- Villeray, Montréal, QC