hi all,
I would like to gather your views about ways of either extending, or abusing the set of Pd atom types, making it possible for pure data objects to pass around arbitrary data. This data would likely be shared with out-of-Pd world. Some ways I am aware of are listed below with comments.
1. Supporting application-specific atom types. There is no such support in Pd, currently.
2. Introducing one new atom type, generic and extendable.
3. Using numbers as pointers or indices.
3a. Plain memory addresses (gridflow?). Validation is possible by placing a header with a magic word at an address. However, representation of an address does not fit into one Pd atom, is clumsy (particularly when maintaining 32/64-bit compatibility), and involves an overhead, possibly significant in some applications.
3b. Indices into a pool (pdp). In order to make validation possible, another selector-atom is required (by looking at an index-atom alone one cannot tell, if sender meant it to be an index). Maintaining a pool is unnecessary burden for some applications.
Neither 3a, nor 3b, is a flexible one-atom solution. Data references have to be wrapped in Pd messages of a special format.
4. Using symbolic references (jitter). This is, basically, a variant of 3b. It requires a special message format.
5. Using fake-symbol tags (upcoming `plustot' external). These are just symbol pointers, but instead of being hashed in the Pd's symbol table, they live purely in an application domain. The application-specific data is attached to the t_symbol `base' structure:
struct _faked { t_symbol f_base; <application-specific-fields> };
where f_base.s_thing and f_base.s_next are always zero, and f_base.s_name is the validation tag.
Fake-symbol pointers may be packed into lists, used in triggers, etc. For any object not aware of their special status, they represent ordinary symbols (except for some malicious objects, like settable receive).
They are serialization-friendly, in the sense, that tags could be registered as providing special hooks for binbuf read and write operations, once such hooks are supported.
Krzysztof
Oh, boy... this raises lots of interesting questions!
A few comments on individual ideas:
- Supporting application-specific atom types. There is no such
support in Pd, currently.
- Introducing one new atom type, generic and extendable.
either of these first two would require adding code to dozens of 'switch' statements both inside and outside Pd. Some code would run slower because of the extra cases to check.
- Using numbers as pointers or indices.
3a. Plain memory addresses (gridflow?). Validation is possible by placing a header with a magic word at an address. However, representation of an address does not fit into one Pd atom, is clumsy (particularly when maintaining 32/64-bit compatibility), and involves an overhead, possibly significant in some applications.
3b. Indices into a pool (pdp). In order to make validation possible, another selector-atom is required (by looking at an index-atom alone one cannot tell, if sender meant it to be an index). Maintaining a pool is unnecessary burden for some applications.
Neither 3a, nor 3b, is a flexible one-atom solution. Data references have to be wrapped in Pd messages of a special format.
- Using symbolic references (jitter). This is, basically,
a variant of 3b. It requires a special message format.
I think 3a isn't really safe. I don't understand why 3b requires anything extra (you can just interpret numbers as indices and let it just be wrong if someone gives you a wrong number.) These _is_ extra work to do to check things when you get a reference to data in a message.
Here's a variant (maybe) of 4: just make a hash table binding symbols to objects. Each separate extension could make its own hash list and maintain its own name space. You can generate symbols "x%d" to name things that the user didn't name.
- Using fake-symbol tags (upcoming `plustot' external). These
are just symbol pointers, but instead of being hashed in the Pd's symbol table, they live purely in an application domain. The application-specific data is attached to the t_symbol `base' structure:
struct _faked { t_symbol f_base; <application-specific-fields> };
where f_base.s_thing and f_base.s_next are always zero, and f_base.s_name is the validation tag.
Fake-symbol pointers may be packed into lists, used in triggers, etc. For any object not aware of their special status, they represent ordinary symbols (except for some malicious objects, like settable receive).
They are serialization-friendly, in the sense, that tags could be registered as providing special hooks for binbuf read and write operations, once such hooks are supported.
That's interesting. The only danger is that someone will want to compare two symbols with the same name (but at different addresses) and get badly confused! Also, if you print a message out and re-read it it will point to a different place; I'm not sure in how many places the "textability" of messages really gets used...
And here's another possibility: 6. somehow overload Pd's "pointer" structure to point to user-definable data structures. This would have the advantage of being compatible with the way atoms are sent around Pd already, and the pointer type already has a hook to protect against stale pointers. The main disadvantage is that the scalar thing is still quite "experimental"!
cheers Miller
On Tue, 11 Nov 2003, Miller Puckette wrote:
- Introducing one new atom type, generic and extendable.
either of these first two would require adding code to dozens of 'switch' statements both inside and outside Pd. Some code would run slower because of the extra cases to check.
jMax does it though. I was quite surprised to see Pd doesn't do it, and I had to perform the following workaround.
- Using numbers as pointers or indices.
3a. Plain memory addresses (gridflow?). Validation is possible by placing a header with a magic word at an address. However, representation of an address does not fit into one Pd atom, is clumsy (particularly when maintaining 32/64-bit compatibility),
I store a 32-bit pointer as two 32-bit floats, which each can store 24-bit unsigned integers exactly (it's guaranteed by IEEE standard).
and involves an overhead, possibly significant in some applications.
It takes a dozen clock cycles or less, which is ~5 ns by today's standards.
I think 3a isn't really safe.
This is definitely not a major cause of crashing with GridFlow. I never mangled those pointers accidentally, and I don't know anyone who'd dare doing it on purpose, as it doesn't buy anyone anything.
I don't understand why 3b requires anything extra (you can just interpret numbers as indices and let it just be wrong if someone gives you a wrong number.)
In my case, I pass pointers I got from malloc, and I can't do range checks on that. If anything, I can use a hashtable of integer keys, for example, or some other sensible data structure, though a hashtable is the most convenient by far.
Each separate extension could make its own hash list and maintain its own name space. You can generate symbols "x%d" to name things that the user didn't name.
BTW, how does one keep reference counts on anonymous data ?
- somehow overload Pd's "pointer" structure to point to
user-definable data structures.
Could you give more info on this?
and the pointer type already has a hook to protect against stale pointers.
What do you mean by that? weakrefs? refcounts?
________________________________________________________________ Mathieu Bouchard http://artengine.ca/matju
hi Miller, thanks so much for your comments,
Miller Puckette wrote: ...
3b. Indices into a pool (pdp). In order to make validation
...
I think 3a isn't really safe. I don't understand why 3b requires anything extra (you can just interpret numbers as indices and let it just be wrong if someone gives you a wrong number.) These _is_
I am confusing things for sure. In general terms, testing that a reference is valid (that a referenced party exists), is one thing, while another thing is testing that what I have is actually a reference, and a proper type of a reference.
Thinking in general terms is mind-boggling, so I am thinking by examples. In a reference-counting scheme currently used in plustot, one can safely assume a reference is valid, after successful testing for type compatibility. This is because a referenced object itself (not a stub) has its refcount incremented when a fake-symbol is being created. The fake-symbol has a refcount too, btw, in order to make sure Pd objects may safely toss it around.
So, in this case, reference validation and type compatibility is the same thing. However, this is a case, in which refcount-controlled object destruction does actually make sense. There are other cases, of course.
For example, the gpointer scheme is different. Glist may be gone, but there is a stub maintained until no reference points there. Thus both tests are needed: is an atom a gpointer, and does it point to an existing glist. (In fact, since glist-type of a gpointer points to glist elements, not to a glist as a whole, there is yet another test needed, for freshness, although it is glist-specific, and not required for other types of a pointee.)
Now, it looks like the gpointer method is more general...
Krzysztof
-
Krzysztof Czaja -
Mathieu Bouchard -
Miller Puckette