moving this to pd-dev....
On 02/03/2015 06:16 PM, katja wrote:
has for using Pd. When working on Pd-double in 2011 I was interested in 'scientific' applications of Pd like impulse response measurement, where double precision is crucial in some calculations. My current focus is more on live performance and I've never felt a need for doubles in this realm. Still I would be happy to help out with
i cannot help thinking of chun's really nice expr~ live-coding set and the problem he experienced with the lack of precision (iirc, his [wrap~] based counters would eventually become inaccurate in timing...which turns into a problem if your minimal techno base drum get's out of sync...)
anyhow.
- graceful handling of binary incompatibility between core <>
externals of different precision
[...]
Regarding binary incompatibility between builds of different precision, this is a serious problem which can't be resolved under the quick-and-easy approach of specifying t_float at compile time. I would
the main problem seems to be, that an external can register a new double-precision object-class to a single-precision pd-runtime (or vice versa, though the former case will lead easier to segfaults).
how about this simple suggestion: a double-precision pd-runtime will only add new classes, if they are registering themselves with "class_new64()" rather than "class_new()". using a preprocessor macro, we can change the "class_new" to "class_new64" in case the user is compiling for double precision. the pd-runtime shall always support class_new64() but throw a warning when called in a single-precision build
something like:
<m_pd.h> EXTERN t_class *class_new32(...) EXTERN t_class *class_new64(...) #if PD_FLOATSIZE == 64 # define class_new class_new64 #else # define class_new class_new32 #endif </m_pd.h>
<m_class.c> t_class*class_new_doit(...) { /* put the actual implementation of the class registry in here */ } t_class*class_new32(...) { #if PD_FLOATSIZE != 32 verbose(1, "not registering single-precision class"); return 0; #endif return class_new_doit(...); } t_class*class_new64(...) { #if PD_FLOATSIZE != 64 verbose(1, "not registering double-precision class"); return 0; #endif return class_new_doit(...); } #undef class_new t_class*class_new(...) { /* legacy way to register (single-precision) classes */ return class_new32(...); } </m_class.c>
afaict, the only thing left is to make sure that Pd handles NULL-pointers as classptrs gracefully in the other class_... functions (class_addbang()) - that is: ignores them.
gfmsadr IOhannes
On 02/03/2015 08:16 PM, IOhannes m zmölnig wrote:
EXTERN t_class *class_new32(...) EXTERN t_class *class_new64(...)
this btw, would also allow to write externals that provide both single- and double-prevision versions of an objectclass, without the need to change the extension.
however, it is not as straightforward as in the "fat" case, where you just glue together multiple (potentially unrelated) binaries of different archs, and the dynamic linker (the OS!) decides which one to use.
something like:
<myclass32.c> #define PD_FLOATSIZE 32 #include <m_pd.h> #include "myclass_impl.c" void myclass_setup32(void) { class_new32(gensym("myclass"),...); } </myclass32.c> <myclass64.c> #define PD_FLOATSIZE 64 #include <m_pd.h> #include "myclass_impl.c" void myclass_setup32(void) { class_new64(gensym("myclass"),...); } </myclass64.c> <myclass.c> void myclass_setup(void) { #ifdef PD_FLOATSIZE == 64 myclass_setup64(); #else myclass_setup32(); #endif } </myclass.c>
it's probably possible to all this in a single wrapper file, but i hope it's easier to understand without preprocessor magic.
gfmdsar IOhannes
On 02/03/2015 08:29 PM, IOhannes m zmölnig wrote:
On 02/03/2015 08:16 PM, IOhannes m zmölnig wrote:
EXTERN t_class *class_new32(...) EXTERN t_class *class_new64(...)
this btw, would also allow to write externals that provide both single- and double-prevision versions of an objectclass, without the need to change the extension.
they key to this is to provide dummy registration classes for the unsupported precisions, so an external compiled for the "wrong" architecture can be loaded by dlopen() but cannot actually *do* anything (wrong).
it's probably possible to all this in a single wrapper file, but i hope it's easier to understand without preprocessor magic.
a probably more readable example is the following: suppose we have a implementation of a given external (whose author used t_float/t_sample throughout, but has not done anything yet to port it to a singe/double "phat" binary. the objectclass is "mycobject" and lives in mycobject.c
then the following should be enough to make a phat binary: <mycobject_phat.c> void mycobject_setup32(); void mycobject_setup64(); void mycobject_setup() { mycobject_setup64(); mycobject_setup32(); } <mycobject_phat.c>
and compile the original code with the above glue as follows: $ cc -c -o mycobject-single.o mycobject.c -DPD_FLOATSIZE=32 \ -Dmycobject_setup=mycobject_setup32 $ cc -c -o mycobject-double.o mycobject.c -DPD_FLOATSIZE=64 \ -Dmycobject_setup=mycobject_setup64 $ cc -c -o mycobject-phat.o mycobject-phat.c $ cc -o mycobject.pd_linux \ mycobject-single.o mycobject-double.o mycobject-phat.o -lc
so the changes required to build such phat binaries are mainly (only¹) in the build-system, and no (properly written) code needs be touched.
gfmadsr IOhannes
¹ the glue-code given above is trivial and it should be easy to write a reuseable wrapper that is part of the build-system. something like embedding the following file in the build-system <phat.c> #define CONCAT_EXPAND(s1, s2) s1 ## s2 #define CONCAT(s1,s2) CONCAT_EXPAND(s1,s2) void CONCAT(SETUP, 32); void CONCAT(SETUP, 64); void SETUP () { CONCAT(SETUP, 32); CONCAT(SETUP, 64); return 0; } </phat.c> and compiling it as (3rd line in the above example): $ cc -c -o mycobject-phat.o phat.c -DSETUP=mycobject_setup
This sounds like a very interesting approach, IOhannes. Regarding disk space it is nowadays no problem at all to store fat binaries, even on a small device like Raspberry Pi.
To be sure if I understand the idea correctly: the setup method of each class is renamed (by the build system, essentially), and precision-aware Pd looks for that new name so it loads the code for the proper precision. This way you can bundle two versions of a class (or lib) in a single executable, one for each precision. The setup method of the wrong precision will not be called by Pd. Am I right?
Does it also mean the rest of the wrong-precision code will not be loaded? What will happen with unintentionally exported symbols, like private functions accidentally not declared static, or functions defined in shared files? Compiler-dependent methods exist to hide symbols by default. Then, the same approach for renaming class setup methods according to precision, might be used to explicitly define them as exported symbol.
What would a legacy Pd build do with fat-precision binaries? Could we consider single precision the default and not rename the setup function compiled for single precision so it will still work with legacy Pd? If the extensions are not modified, a user can't see whether an external is fat-precision or not.
Katja
On Wed, Feb 4, 2015 at 3:40 AM, IOhannes m zmölnig zmoelnig@iem.at wrote:
On 02/03/2015 08:29 PM, IOhannes m zmölnig wrote:
On 02/03/2015 08:16 PM, IOhannes m zmölnig wrote:
EXTERN t_class *class_new32(...) EXTERN t_class *class_new64(...)
this btw, would also allow to write externals that provide both single- and double-prevision versions of an objectclass, without the need to change the extension.
they key to this is to provide dummy registration classes for the unsupported precisions, so an external compiled for the "wrong" architecture can be loaded by dlopen() but cannot actually *do* anything (wrong).
it's probably possible to all this in a single wrapper file, but i hope it's easier to understand without preprocessor magic.
a probably more readable example is the following: suppose we have a implementation of a given external (whose author used t_float/t_sample throughout, but has not done anything yet to port it to a singe/double "phat" binary. the objectclass is "mycobject" and lives in mycobject.c
then the following should be enough to make a phat binary: <mycobject_phat.c> void mycobject_setup32(); void mycobject_setup64(); void mycobject_setup() { mycobject_setup64(); mycobject_setup32(); } <mycobject_phat.c>
and compile the original code with the above glue as follows: $ cc -c -o mycobject-single.o mycobject.c -DPD_FLOATSIZE=32 \ -Dmycobject_setup=mycobject_setup32 $ cc -c -o mycobject-double.o mycobject.c -DPD_FLOATSIZE=64 \ -Dmycobject_setup=mycobject_setup64 $ cc -c -o mycobject-phat.o mycobject-phat.c $ cc -o mycobject.pd_linux \ mycobject-single.o mycobject-double.o mycobject-phat.o -lc
so the changes required to build such phat binaries are mainly (only¹) in the build-system, and no (properly written) code needs be touched.
gfmadsr IOhannes
¹ the glue-code given above is trivial and it should be easy to write a reuseable wrapper that is part of the build-system. something like embedding the following file in the build-system <phat.c> #define CONCAT_EXPAND(s1, s2) s1 ## s2 #define CONCAT(s1,s2) CONCAT_EXPAND(s1,s2) void CONCAT(SETUP, 32); void CONCAT(SETUP, 64); void SETUP () { CONCAT(SETUP, 32); CONCAT(SETUP, 64); return 0; } </phat.c> and compiling it as (3rd line in the above example): $ cc -c -o mycobject-phat.o phat.c -DSETUP=mycobject_setup
Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-02-04 10:57, katja wrote:
To be sure if I understand the idea correctly: the setup method of each class is renamed (by the build system, essentially), and precision-aware Pd looks for that new name so it loads the code for the proper precision. This way you can bundle two versions of a class (or lib) in a single executable, one for each precision. The setup method of the wrong precision will not be called by Pd. Am I right?
no. we might misunderstand each other.
the basic idea (without phat binaries) ====
- - Pd only looks for the good olde <name>_setup() function (or it's hexmunged equivalent). - - the external registers a new object-class with a precision-aware callback function (*class_new* for single-precision; *class_new64* for double precision); class_new64() is used automatically due to a define in m-pd.h triggered by a double-precision build.
loading scenarios: * when an external is loaded, the Pd-runtime calls the external *setup* function (via *dlopen()*).
* the external then registers it's objectclasses with the pd-runtime
** nothing really changes for a single-precision Pd-runtime (including a legacy Pd-runtime) loading a single-precision external.
** when a double-precision external is loaded by a double-precision runtime, the objectclasses will be registered via the new callback "class_new64()".
** when a single-precision external is loaded by a double-precision runtime, it will register the objectclasses via class_new() which is a noop (probably giving the user a warning about mismatched precisions)
** when a double-precision external is loaded by a (new) single-precision runtime, it will register the objectclasses via class_new64() which is a noop (probably giving the user a warning about mismatched precisions)
** when a double-precision external is loaded by a legacy (single-precision) runtime, it would register the objectclasses via class_new64() which does not exist (and thus the dlopen()ing ofthe external won't work in the first place)
multi-precision libraries (what i call "phat binaries" for lack of a better work; it's contains both "fat" and "precision") ===
- - the external's setup()-function calls both class_new() and class_new64() to register two different specializations of the same object (i just borrowed the term "specialization" from C++-templates)
- - a (new) single-precision Pd will discard the class_new64() call, and thus only *see* the single-precision objectclass.
- - a (new) double-precision Pd will discard the class_new() call, and thus only *see* the double-precision objectclass.
- - a legacy (single-precision) Pd will not be able to load the phat binary, since the "class_new64" symbol is missing.
the last item is obviously a little problem, but that problem always exists if an external uses a newly introduced function (e.g. using "logpost()" will make the external unusable on Pd<=0.42)
Does it also mean the rest of the wrong-precision code will not be loaded? What will happen with unintentionally exported symbols, like private functions accidentally not declared static, or functions defined in shared files? Compiler-dependent methods exist to hide symbols by default. Then, the same approach for renaming class setup methods according to precision, might be used to explicitly define them as exported symbol.
good points. my approach assumed, that the code was written "correctly" ("proper"). that is: t_float/t_sample is used throughout; *all* functions that need not be exported are not exported. probably others.
hiding (aka: not-exporting) symbols by default won't help us much, as this only works at the dylib boundaries: within the entire phat-binary the symbols are still visible (so it's different from "static" functions).
anyhow: my point was mainly, that it should be fairly simple to create phat binaries of many (simplistic) externals (and i think most externals fall into this category). for more complex externals, a lot of manual work might still be involved,
What would a legacy Pd build do with fat-precision binaries? Could we consider single precision the default and not rename the setup function compiled for single precision so it will still work with legacy Pd? If the extensions are not modified, a user can't see whether an external is fat-precision or not.
see my other (short) mail (and the above explanations): single-precision externals should continue to use the olde names. if we want to support phat binaries in legacy runtimes, we probably should think again (adding an extension will not help us here either). at least on linux (and most likely on OSX; but more unlikely on W32) it should be possible to provide a "stub-external", that only exports the new "class_new64" symbol (as a noop). having this stub-external loaded, should allow to dlopen() a phat binary (or a double-precision binary) with the double-precision parts not doing any harm.
fgmasdr IOhannes
Sorry IOhannes, my previous mail was sent right before your explanation saying where I got it wrong. So please ignore it.
Katja
On Wed, Feb 4, 2015 at 11:51 AM, IOhannes m zmoelnig zmoelnig@iem.at wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-02-04 10:57, katja wrote:
To be sure if I understand the idea correctly: the setup method of each class is renamed (by the build system, essentially), and precision-aware Pd looks for that new name so it loads the code for the proper precision. This way you can bundle two versions of a class (or lib) in a single executable, one for each precision. The setup method of the wrong precision will not be called by Pd. Am I right?
no. we might misunderstand each other.
the basic idea (without phat binaries)
- Pd only looks for the good olde <name>_setup() function (or it's
hexmunged equivalent).
- the external registers a new object-class with a precision-aware
callback function (*class_new* for single-precision; *class_new64* for double precision); class_new64() is used automatically due to a define in m-pd.h triggered by a double-precision build.
loading scenarios:
- when an external is loaded, the Pd-runtime calls the external
*setup* function (via *dlopen()*).
- the external then registers it's objectclasses with the pd-runtime
** nothing really changes for a single-precision Pd-runtime (including a legacy Pd-runtime) loading a single-precision external.
** when a double-precision external is loaded by a double-precision runtime, the objectclasses will be registered via the new callback "class_new64()".
** when a single-precision external is loaded by a double-precision runtime, it will register the objectclasses via class_new() which is a noop (probably giving the user a warning about mismatched precisions)
** when a double-precision external is loaded by a (new) single-precision runtime, it will register the objectclasses via class_new64() which is a noop (probably giving the user a warning about mismatched precisions)
** when a double-precision external is loaded by a legacy (single-precision) runtime, it would register the objectclasses via class_new64() which does not exist (and thus the dlopen()ing ofthe external won't work in the first place)
multi-precision libraries (what i call "phat binaries" for lack of a better work; it's contains both "fat" and "precision") ===
- the external's setup()-function calls both class_new() and
class_new64() to register two different specializations of the same object (i just borrowed the term "specialization" from C++-templates)
- a (new) single-precision Pd will discard the class_new64() call, and
thus only *see* the single-precision objectclass.
- a (new) double-precision Pd will discard the class_new() call, and
thus only *see* the double-precision objectclass.
- a legacy (single-precision) Pd will not be able to load the phat
binary, since the "class_new64" symbol is missing.
the last item is obviously a little problem, but that problem always exists if an external uses a newly introduced function (e.g. using "logpost()" will make the external unusable on Pd<=0.42)
Does it also mean the rest of the wrong-precision code will not be loaded? What will happen with unintentionally exported symbols, like private functions accidentally not declared static, or functions defined in shared files? Compiler-dependent methods exist to hide symbols by default. Then, the same approach for renaming class setup methods according to precision, might be used to explicitly define them as exported symbol.
good points. my approach assumed, that the code was written "correctly" ("proper"). that is: t_float/t_sample is used throughout; *all* functions that need not be exported are not exported. probably others.
hiding (aka: not-exporting) symbols by default won't help us much, as this only works at the dylib boundaries: within the entire phat-binary the symbols are still visible (so it's different from "static" functions).
anyhow: my point was mainly, that it should be fairly simple to create phat binaries of many (simplistic) externals (and i think most externals fall into this category). for more complex externals, a lot of manual work might still be involved,
What would a legacy Pd build do with fat-precision binaries? Could we consider single precision the default and not rename the setup function compiled for single precision so it will still work with legacy Pd? If the extensions are not modified, a user can't see whether an external is fat-precision or not.
see my other (short) mail (and the above explanations): single-precision externals should continue to use the olde names. if we want to support phat binaries in legacy runtimes, we probably should think again (adding an extension will not help us here either). at least on linux (and most likely on OSX; but more unlikely on W32) it should be possible to provide a "stub-external", that only exports the new "class_new64" symbol (as a noop). having this stub-external loaded, should allow to dlopen() a phat binary (or a double-precision binary) with the double-precision parts not doing any harm.
fgmasdr IOhannes
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1
iQIcBAEBCAAGBQJU0fnBAAoJELZQGcR/ejb4ZvsQAIq1zomPlsirrBGFE7oVIc1p qPecDk9zReG1E/wSyU6ntLNasXUAojd44eTo4acijH86RR7n+MRQB5RgAAhmf1mm ZVesL8R7AoJ1qYkCg8AfkgSkYD+GZWydebfiEHTiWks4fq7amzYWeHk8BENMHCAU 4f7j2N1bp66E1GQTdhS/dI/Rm39Q/8Ja8M7lrI2nYbQx0WzGBYFV9hMkuYGO96zq aCNb3+UCMwTUDfWRFnyeGxTystOOB2kuaxPDKivYmpduX7Anfryb/IaypkVBxRoQ goU2b/FhglKyiNtlQRLqDwbxSgMD84rfwxFnQbGyRA79ZQHEi+riYB0l9ayrO3pM PwDKcfMIBA09awtWlv+a7uYXrQgK44nZcSoHkvZgGuZmQobEAZIwWKw8VgO+H5Di D/XoUDmXGPsHnElSc1kSMFwk0jIXZb2QaVQAK3fy4xvAZ+wWbzcPdS5GdF4rw0QR rXyjXMRY84G+3j+a5X4pByVaR6LFFDGaacnKE8hltdYhFhDhbrH6PQ/jn/UJGAMG JY+44knoIJQPlnXrWzsn56VnVXet8uE4SG1XSIueSKZpvwVJ42aYmStXZHJygk/r eQ0EMZ2YzIRbssCux5OYM7Oos/XOT2nSqE6aAFJACMJ2j++pEJUspOZwkC2DJc2I 31iJOloztJrgoa3z++RI =ZJt5 -----END PGP SIGNATURE-----
Pd-dev mailing list Pd-dev@lists.iem.at http://lists.puredata.info/listinfo/pd-dev
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-02-04 10:57, katja wrote:
What would a legacy Pd build do with fat-precision binaries? Could we consider single precision the default and not rename the setup function compiled for single precision so it will still work with legacy Pd? If the extensions are not modified, a user can't see whether an external is fat-precision or not.
just a word with my debian maintainer hat on:
if Pd-vanilla supported double-precision builds, i would like to make these available for Debian and derivatives as well.
if i made single- and double-precision builds co-installable (which iÄm not sure yet), it would be as "pd" and "pd64", with pd64 using different default configuration (~/.pd64settings) and search paths (/usr/lib/pd64/extra/). and install all (packaged) externals that can be build for double precision into /usr/lib/pd64/extra/, rather than having phat builds.
i don't think we should spend too much time/energy on designing and supporting foolproof phat builds at all (as this is probably impossible without the help from the OS).
and the extensions in Pd are already a big mess; any other environment i know that supports loading of binary plugins, uses the OS's default extensions: .so, .dylib, .dll; using non-standard extensions is often *very* inconvenient...so please do not add another one.
fgamsd IOhannes
On Wed, Feb 4, 2015 at 12:03 PM, IOhannes m zmoelnig zmoelnig@iem.at wrote:
just a word with my debian maintainer hat on:
if Pd-vanilla supported double-precision builds, i would like to make these available for Debian and derivatives as well.
if i made single- and double-precision builds co-installable (which iÄm not sure yet), it would be as "pd" and "pd64", with pd64 using different default configuration (~/.pd64settings) and search paths (/usr/lib/pd64/extra/). and install all (packaged) externals that can be build for double precision into /usr/lib/pd64/extra/, rather than having phat builds.
i don't think we should spend too much time/energy on designing and supporting foolproof phat builds at all (as this is probably impossible without the help from the OS).
Allright, I was just carried away by the prospect of phat / fat-precision binaries. But indeed it's regular practice to keep Pd build and externals together. So the only critical point is install time. If users add external libs in the path of Pd 'by hand' they need to verify the origin anyway. For example one can't add Pd-L2Ork externals to Pd vanilla / extended or vice versa, even though these have the same name and extension.
So what you propose (for the 'non-phat' case) is a protection mechanism to avoid Pd loading a wrong-precision external. That is most important, because a wrong-precision external causes spurious malfunctions, of which segfaulting may not even be the worst because that is at least a clear signal. And a double precision Pd build and process should identify itself as such so you need not guess what precision you're using at any moment. It is perfectly possible to run Pd instances of different precision simultaneously, is what I recall from my experiments in 2011.
Katja
On Wed, Feb 04, 2015 at 12:44:24PM +0100, katja wrote:
On Wed, Feb 4, 2015 at 12:03 PM, IOhannes m zmoelnig zmoelnig@iem.at wrote:
just a word with my debian maintainer hat on:
if Pd-vanilla supported double-precision builds, i would like to make these available for Debian and derivatives as well.
if i made single- and double-precision builds co-installable (which iÃm not sure yet), it would be as "pd" and "pd64", with pd64 using different default configuration (~/.pd64settings) and search paths (/usr/lib/pd64/extra/). and install all (packaged) externals that can be build for double precision into /usr/lib/pd64/extra/, rather than having phat builds.
i don't think we should spend too much time/energy on designing and supporting foolproof phat builds at all (as this is probably impossible without the help from the OS).
Allright, I was just carried away by the prospect of phat / fat-precision binaries. But indeed it's regular practice to keep Pd build and externals together. So the only critical point is install time. If users add external libs in the path of Pd 'by hand' they need to verify the origin anyway. For example one can't add Pd-L2Ork externals to Pd vanilla / extended or vice versa, even though these have the same name and extension.
So what you propose (for the 'non-phat' case) is a protection mechanism to avoid Pd loading a wrong-precision external. That is most important, because a wrong-precision external causes spurious malfunctions, of which segfaulting may not even be the worst because that is at least a clear signal. And a double precision Pd build and process should identify itself as such so you need not guess what precision you're using at any moment. It is perfectly possible to run Pd instances of different precision simultaneously, is what I recall from my experiments in 2011.
Katja
It might be that just building on to the current setup would be least troublesome. In distributions such as the debian packages, everybody could revert to .dll, .pd_darwin, and .pd_linux (which is OK because there would only be one architecture and one float-size per distro) and then a single layer of ad-hoc extents for anyone wanting to distribute fat externs (by which I mean, externs in directories containing .l_arm, .l_i386, .d_ppc, and the rest of the carnival, now to include new ones like .l_ia64_double or something.
As it does now, Pd would search first for the more specific one, then fall back to the more general one.
And, if it's useful, going forward Pd could even add a third possibility which would be to search for .so (linux) or .dylib (mac I think). Why not...
cheers Miller
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-02-03 20:16, IOhannes m zmölnig wrote:
t_class*class_new32(...) {
forget *class_new32()* the single-precision class-registration callback function shall keep its name *class_new()* so an old (single-precision only) version of the Pd-runtime can still load a (single-precision) external.
loading a double-precision external with a legacy (single-precision) Pd-runtime will fail due to unresolved symbols.
fgmasdr IOhannes
I like the term 'fat-precision'. So if we have fat-precision binaries where single and double precision versions of an external class or lib are linked together, 'legacy' (well, current) Pd would refuse to load the single precision part, because it comes across a function <myclass>_setup64() which is unresolved. Unresolved means it can't find the definition, right? And we can't provide a dummy definition for this situation, as that would get in the way of the definition in precision-aware Pd. Hmm. This defies the purpose of fat-precision binaries, where Pd users don't need to bother about precisionness of externals.
Katja
-
IOhannes m zmoelnig -
IOhannes m zmölnig
-
katja -
Miller Puckette