Forgive me if this has been gone over, but what's the behavior if a single-precision Pd tries to load a double-precision external or vice versa? Does it fail to load or simply crash?
If it crashes, maybe there needs to be some mechanism to query the compiled precision of the external, ie. some sort of function pointer or define, etc. If it's not there, then assume single-precision and act accordingly. It would be nice to have useful error print concerning this. I'm just musing here. :)
On Feb 26, 2020, at 1:58 PM, pd-dev-request@lists.iem.at wrote:
Message: 4 Date: Wed, 26 Feb 2020 13:58:50 +0100 From: Christof Ressi <info@christofressi.com mailto:info@christofressi.com> To: pd-dev@lists.iem.at mailto:pd-dev@lists.iem.at Subject: Re: [PD-dev] [PD] double precision merged Message-ID: <7f65121f-810d-c952-40f8-4f0facbbb1a3@christofressi.com mailto:7f65121f-810d-c952-40f8-4f0facbbb1a3@christofressi.com> Content-Type: text/plain; charset=utf-8; format=flowed
I see you point and I think it's a philosophical issue. In Supercollider, for example, I can't compile a UGen plugin and expect it to run on both Scsynth and Supernova, I rather have to pass the correct define ("SUPERNOVA"). Plugins are therefore usually built twice - with and without "-DSUPERNOVA" - and since they have different extensions and export different symbols, they can coexist. I think this could be a solution for Pd as well. If we had some naming convention for double precision externals, we can then just built both versions unconditionally and Pd will load the correct version. This can be automated by pd-lib-builder.
Christof
-------- Dan Wilcox @danomatika http://twitter.com/danomatika danomatika.com http://danomatika.com/ robotcowboy.com http://robotcowboy.com/
Em qua., 26 de fev. de 2020 às 10:09, Dan Wilcox danomatika@gmail.com escreveu:
Forgive me if this has been gone over, but what's the behavior if a single-precision Pd tries to load a double-precision external or vice versa? Does it fail to load or simply crash?
If it crashes, maybe there needs to be some mechanism to query the compiled precision of the external, ie. some sort of function pointer or define, etc. If it's not there, then assume single-precision and act accordingly. It would be nice to have useful error print concerning this. I'm just musing here. :)
I've tested loading externals and it just doesn't create and complain, but trying to load cyclone with [declare -lib cyclone] crashes Pd!
On 26.02.20 14:12, Alexandre Torres Porres wrote:
I've tested loading externals and it just doesn't create and complain, but trying to load cyclone with [declare -lib cyclone] crashes Pd!
could that be a duplicate of https://lists.puredata.info/pipermail/pd-dev/2019-12/022222.html?
fgmasdf IOhannes
On 26.02.20 14:22, IOhannes m zmoelnig wrote:
On 26.02.20 14:12, Alexandre Torres Porres wrote:
I've tested loading externals and it just doesn't create and complain, but trying to load cyclone with [declare -lib cyclone] crashes Pd!
could that be a duplicate of https://lists.puredata.info/pipermail/pd-dev/2019-12/022222.html?
ah sure:
class_new() *might* return a NULL-pointer (and it will, if the precision doesn't match), so you have to check for that before you access any class members.
asdrm IOhannes
On 2/26/20 2:28 PM, IOhannes m zmoelnig wrote:
class_new() *might* return a NULL-pointer (and it will, if the precision doesn't match), so you have to check for that before you access any class members.
for the sake of seeing how many packages are impacted by this problem, i've done some testing.
i downloaded the newest version available on deken for each package that has a binary available for my platform (Linux/amd64), and attempted to load the binaries in there with a double-precision Pd.
in 24 of these packages there was at least one binary that crashed Pd (see attachment).
two notes: - binaries that did not load (not even on a single-precision Pd) were not tested. this includes (some, but not necessary all) binaries from the following packages: Jamoma, extra, hcs, iem16, libdir, mrpeach, ofelia, pdp, pix_drum, pix_fiducialtrack, pix_mano, purest_json, tclpd, unauthorized, vanilla - most externals from thomas grill crash, which suggests a problem with flext.
gmards IOhannes
Hi Iohannes,
thanks for bringing up double-precision problems with flext.
i downloaded the newest version available on deken for each package that has a binary available for my platform (Linux/amd64), and attempted to load the binaries in there with a double-precision Pd.
i have not followed the double precision developments lately, and all the flext binaries are traditional single-precision compiles.
Before i spend time debugging the issue: Are single-precision externals meant to interoperate with Pd double precision without changes? If not, how are single and double precision externals separately identified?
thanks, Thomas
-- Thomas Grill http://grrrr.org
On 3/1/20 10:51 PM, IOhannes m zmölnig wrote:
On 2/26/20 2:28 PM, IOhannes m zmoelnig wrote:
class_new() *might* return a NULL-pointer (and it will, if the precision doesn't match), so you have to check for that before you access any class members.
TL;DR there was a bug in Pd that created a few false positives in my original list.
so while i was accusing externals of not taking care of NULL-pointers properly, the code that would print the nice error message when trying to load an external with the wrong float-size would do exactly the same.
fixing this (as in 0cc9869dd in the update/0.50 branch on github), reduces the number of libraries that crash when being loaded by two (2).
the false positives were "else" and "iemguts". other libraries (like Gem) still crash, but now they are crashing somewhere else... so the list of crashing packages is now:
- Gem - absattr - clk - constantq~ - context - creb - cyclone - hcs - ml.lib - moonlib - nilwind - ossia - pddp - pdlua - pof - pool - py - unauthorized - vasp - vinylcontrol~ - xsample - zconf
gfmasdr IOhannes
On 3/2/20 10:43 AM, IOhannes m zmölnig wrote:
On 3/1/20 10:51 PM, IOhannes m zmölnig wrote:
On 2/26/20 2:28 PM, IOhannes m zmoelnig wrote:
class_new() *might* return a NULL-pointer (and it will, if the precision doesn't match), so you have to check for that before you access any class members.
TL;DR there was a bug in Pd that created a few false positives in my original list.
since all flext externals are crashing, i thought i had a look at those specifically. it turns out, that flext has it's own use for NULL-classes, as it internally uses them to create a persistent (and invisible) "flext buffer helper" object.
with mismatched float-sizes, this results in calling `pd_new(NULL)` - something that never happens in the ordinary flow, where pd_new() only gets called in the newmethod of an objectclass. Pd itself doesn't call the newmethod of objectclasses with mismatched float-size, so this is usually not a problem.
however, it's easy enough to fix. pd_new() already does a NULL-ptr check, but only prints an error and happily proceeds to access it's members, leading to the inevitable crash. doing an early exit in this case, reduces the list of crashing externals to the following eleven (11) packages: - creb - cyclone - hcs - moonlib - nilwind - ossia - pddp - pdlua - pof - unauthorized - Gem (crash fixed upstream, but no release yet) - py (this is a false-positive, as it also crashes if the float-sizes do match) - context (the crash is triggered by the embedded cyclone)
even though, we can fix pd_new() to accept NULL-classes, flext should probably handle this explicitly and not attempt to create the "flext buffer helper" object if the class-pointer is invalid.
gfadsmr IOhannes
Hi all, thanks to IOhannes for debugging. I have pushed fixes to https://github.com/grrrr/flext which should make flext (and flext-based objects) fully compatible with 64-bit DSP, that is, flext-based project should compile and run (or at least, not crash).
My question is now how to separate 32-bit DSP from 64-bit DSP binaries. There doesn't seem a separate file extension, right? How are two variants handled in deken?
best, Thomas
Am 02.03.2020 um 11:51 schrieb IOhannes m zmölnig zmoelnig@iem.at:
Signierter PGP-Teil On 3/2/20 10:43 AM, IOhannes m zmölnig wrote:
On 3/1/20 10:51 PM, IOhannes m zmölnig wrote:
On 2/26/20 2:28 PM, IOhannes m zmoelnig wrote:
class_new() *might* return a NULL-pointer (and it will, if the precision doesn't match), so you have to check for that before you access any class members.
TL;DR there was a bug in Pd that created a few false positives in my original list.
since all flext externals are crashing, i thought i had a look at those specifically. it turns out, that flext has it's own use for NULL-classes, as it internally uses them to create a persistent (and invisible) "flext buffer helper" object.
with mismatched float-sizes, this results in calling `pd_new(NULL)` - something that never happens in the ordinary flow, where pd_new() only gets called in the newmethod of an objectclass. Pd itself doesn't call the newmethod of objectclasses with mismatched float-size, so this is usually not a problem.
however, it's easy enough to fix. pd_new() already does a NULL-ptr check, but only prints an error and happily proceeds to access it's members, leading to the inevitable crash. doing an early exit in this case, reduces the list of crashing externals to the following eleven (11) packages:
- creb
- cyclone
- hcs
- moonlib
- nilwind
- ossia
- pddp
- pdlua
- pof
- unauthorized
- Gem (crash fixed upstream, but no release yet)
- py (this is a false-positive, as it also crashes if the float-sizes do
match)
- context (the crash is triggered by the embedded cyclone)
even though, we can fix pd_new() to accept NULL-classes, flext should probably handle this explicitly and not attempt to create the "flext buffer helper" object if the class-pointer is invalid.
gfadsmr IOhannes
-- Thomas Grill http://grrrr.org
On 03.03.20 14:55, Thomas Grill wrote:
Hi all, thanks to IOhannes for debugging. I have pushed fixes to https://github.com/grrrr/flext which should make flext (and flext-based objects) fully compatible with 64-bit DSP, that is, flext-based project should compile and run (or at least, not crash).
My question is now how to separate 32-bit DSP from 64-bit DSP binaries. There doesn't seem a separate file extension, right?
right. there's an ongoing discussion at [900]
the gist is:
## current situation pd-double uses "class_new64()" instead of "class_new()" (which is reserved for registering sijngle-precision externals). if a (single-precision) external calls "class_new()" in a double-precision runtime, it will spit out a warning and return a NULL-pointer. vice versa, if a (double-precision) external calls "class_new64()" in a single-precision runtime, it will spit out a warning and return a NULL-pointer. this is to protect externals from being loaded into a runtime of the wrong floatsize. to allow simple compilation of double-precision externals, "class_new" is redefined (in m_pd.h) to "class_new64" if PD_FLOATSIZE==64.
i personally am convinced that this is a *required* protection of the runtime.
## TODO however, people are not entirely happy yet ([900]). there's basically two classes of problems: - distribution of packages that contain binaries for both float-sizes - backward compatibility
### packages with both float-sizes the current implementation doesn't solve the problem on how to distribute binaries (e.g. via deken) either side-by-side (multiple binary files) or in a single phat binary (that containes code for both precisions).
the former would require a way to distinguish the filenames based on the precision (that is: a filename-extension schema). people have proposed something like "<library>.l_amd64-DP", which i personally find rather ugly. a somewhat nicer schema might be "<library>.float64.l_amd64" (which i would prefer, as it keeps the final extension intact)
the latter (phat binaries that are compiled for both double- and single-precision) is theoretically possible, but people have to jump through a lot of hoops to make it work. this should, and probably could be made simpler. see [1] for a suggestion to mangle the entry-point (aka setup() function) of an external. this could make compilation of phat externals much simpler...iff the only public functions are the setup-functions.
### backward compatibility the other problem is, that binaries compiled for double-precision use a new symbol ("class_new64") which is not present in older versions of Pd (Pd<<0.51). This is of course not a problem for double-precision only binaries (which you don't want to load anyhow in the olde single-precision runtime). but if the binary contains code for *both* single- abd double-precision, then you might expect to be able to load the single-precision variant in legacy Pd's.
i think this can be solved by weak linking (aka: "optional symbols" that are set to NULL if not available). i have an experimental branch at [2] where i toyed with this idea today, and so far it seems to work nicely. the biggest caveat is, that not all OSs support weak linking, and of course MicroSoft Windows does not. so my branch uses a little extra magic to implement a replacement using GetProcAddress(), but that requires to include <windows.h>, which somewhat bloats m_pd.h
How are two variants handled in deken?
deken (the cmdline tool) currently inspects the binary to see whether it imports the "class_new" symbol (in which case it is a single-precision binary) resp the "class_new64" symbol (in which case it is a double-precision binary). if both symbols are imported, then the binary is multi-precision. this information is encoded into the filename, e.g. a deken package "foobar[v1.2.3](Linux-ppc-64)(Linux-ppc-32).dek" contains PowerPC binaries for both single and double-precision.
the deken-plugin (the downloader/installer) just uses the the filename to determine whether a given search-result matches it's own architecture/precision.
gmasdr IOhannes
[900 ]https://github.com/pure-data/pure-data/issues/900 [1] https://lists.puredata.info/pipermail/pd-dev/2019-12/022214.html [2] https://git.iem.at/pd/pure-data/-/tree/tests/double-API
+1
"<library>.float64.l_amd64"
The more we can tell about a binary by just reading its name the better.
Mensaje telepatico asistido por maquinas.
Dear all, i have published binaries on deken for many of my externals covering a lot of platforms and variants (i386/x86_64/arm6/arm7/arm64 on darwin/windows/linux, single and double precision floats). Specifically, binaries for Windows and Pd double precision have been newly added. I am not able to test the externals on all platforms. If you like, please give them a try and let me know should there be crashes or other problems. best, Thomas
Am 03.03.2020 um 14:55 schrieb Thomas Grill gr@grrrr.org:
Hi all, thanks to IOhannes for debugging. I have pushed fixes to https://github.com/grrrr/flext which should make flext (and flext-based objects) fully compatible with 64-bit DSP, that is, flext-based project should compile and run (or at least, not crash).
My question is now how to separate 32-bit DSP from 64-bit DSP binaries. There doesn't seem a separate file extension, right? How are two variants handled in deken?
best, Thomas
Am 02.03.2020 um 11:51 schrieb IOhannes m zmölnig zmoelnig@iem.at:
Signierter PGP-Teil On 3/2/20 10:43 AM, IOhannes m zmölnig wrote:
On 3/1/20 10:51 PM, IOhannes m zmölnig wrote:
On 2/26/20 2:28 PM, IOhannes m zmoelnig wrote:
class_new() *might* return a NULL-pointer (and it will, if the precision doesn't match), so you have to check for that before you access any class members.
TL;DR there was a bug in Pd that created a few false positives in my original list.
since all flext externals are crashing, i thought i had a look at those specifically. it turns out, that flext has it's own use for NULL-classes, as it internally uses them to create a persistent (and invisible) "flext buffer helper" object.
with mismatched float-sizes, this results in calling `pd_new(NULL)` - something that never happens in the ordinary flow, where pd_new() only gets called in the newmethod of an objectclass. Pd itself doesn't call the newmethod of objectclasses with mismatched float-size, so this is usually not a problem.
however, it's easy enough to fix. pd_new() already does a NULL-ptr check, but only prints an error and happily proceeds to access it's members, leading to the inevitable crash. doing an early exit in this case, reduces the list of crashing externals to the following eleven (11) packages:
- creb
- cyclone
- hcs
- moonlib
- nilwind
- ossia
- pddp
- pdlua
- pof
- unauthorized
- Gem (crash fixed upstream, but no release yet)
- py (this is a false-positive, as it also crashes if the float-sizes do
match)
- context (the crash is triggered by the embedded cyclone)
even though, we can fix pd_new() to accept NULL-classes, flext should probably handle this explicitly and not attempt to create the "flext buffer helper" object if the class-pointer is invalid.
gfadsmr IOhannes
-- Thomas Grill http://grrrr.org
-- Thomas Grill http://grrrr.org
Hi Thomas,
The only thing I got from deken (for win64) was "purest_json[v1.4.3](Windows-amd64-64).dek"
Pd-double refuses to load it complaining:
"refusing to load 32bit-float object 'json-encode' into 64bit-float Pd"
Did you compiled it against a Pd-double?
In case you need a Pd-double compiled for win64 here is one:
The list think this link is spam so I'll try in an attached zip (just a txt with the link). see attached.
[1] I'm running a self-hosted instance of "Nextcloud" on an asus eeepc 1GB-ram Intel Atom 32-bit 1.6-Ghz with debian/buster with an ADSL connection.
Mensaje telepatico asistido por maquinas.
On 3/6/2020 12:33 PM, Thomas Grill wrote:
Dear all, i have published binaries on deken for many of my externals covering a lot of platforms and variants (i386/x86_64/arm6/arm7/arm64 on darwin/windows/linux, single and double precision floats). Specifically, binaries for Windows and Pd double precision have been newly added. I am not able to test the externals on all platforms. If you like, please give them a try and let me know should there be crashes or other problems. best, Thomas
sorry the txt was empty.
One more try.
see attached.
Mensaje telepatico asistido por maquinas.
On 3/6/2020 6:21 PM, Lucas Cordiviola wrote:
Hi Thomas,
The only thing I got from deken (for win64) was "purest_json[v1.4.3](Windows-amd64-64).dek"
Pd-double refuses to load it complaining:
"refusing to load 32bit-float object 'json-encode' into 64bit-float Pd"
Did you compiled it against a Pd-double?
In case you need a Pd-double compiled for win64 here is one:
The list think this link is spam so I'll try in an attached zip (just a txt with the link). see attached.
[1] I'm running a self-hosted instance of "Nextcloud" on an asus eeepc 1GB-ram Intel Atom 32-bit 1.6-Ghz with debian/buster with an ADSL connection.
Mensaje telepatico asistido por maquinas.
On 3/6/2020 12:33 PM, Thomas Grill wrote:
Dear all, i have published binaries on deken for many of my externals covering a lot of platforms and variants (i386/x86_64/arm6/arm7/arm64 on darwin/windows/linux, single and double precision floats). Specifically, binaries for Windows and Pd double precision have been newly added. I am not able to test the externals on all platforms. If you like, please give them a try and let me know should there be crashes or other problems. best, Thomas
Am 6. März 2020 22:21:53 MEZ schrieb Lucas Cordiviola lucarda27@hotmail.com:
Hi Thomas,
The only thing I got from deken (for win64) was "purest_json[v1.4.3](Windows-amd64-64).dek"
Pd-double refuses to load it complaining:
"refusing to load 32bit-float object 'json-encode' into 64bit-float Pd"
Did you compiled it against a Pd-double?
purest_json is from *another* thomas. also i guess that the deken-name (indicating the precision) is simply wrong.
mfg.hft.fsl IOhannes
I've tested loading externals and it just doesn't create and complain
Are you sure? Turn on log level 4. Here's what I get when I try to load a single precision [markex/randomF] in my double precision built:
refusing to load 32bit-float object 'randomF' into 64bit-float Pd maximum object loading depth 1000 reached markex/randomF
Note that actual warning is only visible with log level 3 and above.
but trying to load cyclone with [declare -lib cyclone] crashes Pd!
Please run Pd in a debugger and see where it crashes.
I also get a crash when trying to load [iemguts/receivecanvas]. In this specific case, the reason is that this object registers a proxy class without a name (passing NULL instead of a symbol for the class name), so Pd segfaults when it tries to print the warning. Pd should do a NULL check and probably omit the warning for unnamed classes.
Christof
On 26.02.2020 14:12, Alexandre Torres Porres wrote:
Em qua., 26 de fev. de 2020 às 10:09, Dan Wilcox <danomatika@gmail.com mailto:danomatika@gmail.com> escreveu:
Forgive me if this has been gone over, but what's the behavior if a single-precision Pd tries to load a double-precision external or vice versa? Does it fail to load or simply crash? If it crashes, maybe there needs to be some mechanism to query the compiled precision of the external, ie. some sort of function pointer or define, etc. If it's not there, then assume single-precision and act accordingly. It would be nice to have useful error print concerning this. I'm just musing here. :)I've tested loading externals and it just doesn't create and complain, but trying to load cyclone with [declare -lib cyclone] crashes Pd!
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
On 26.02.20 14:26, Christof Ressi wrote:
I've tested loading externals and it just doesn't create and complain
Are you sure? Turn on log level 4.
i think alex really meant:
"I've tested loading externals and it just doesn't create[,] and [instead] complain[s]."
mgfsdr IOhannes
Em qua., 26 de fev. de 2020 às 10:30, IOhannes m zmoelnig zmoelnig@iem.at escreveu:
i think alex really meant: "I've tested loading externals and it just doesn't create[,] and [instead] complain[s]."
yup
On 26.02.20 14:09, Dan Wilcox wrote:
Forgive me if this has been gone over, but what's the behavior if a single-precision Pd tries to load a double-precision external or vice versa? Does it fail to load or simply crash?
with Pd>=0.51, the class_new will refuse to register the objectlcass (and instead return NULL).
e.g.
refusing to load 32bit-float object 'choice' into 64bit-float Pd
resp.
refusing to load 64bit-float object 'choice' into 32bit-float Pd
if you attempt to load a double-precision external with Pd<0.51, it will refuse to load the library with an error:
choice.pd_linux: undefined symbol: class_new64
fgmasdr IOhannes
PS: https://lists.puredata.info/pipermail/pd-dev/2015-02/020073.html
-
Alexandre Torres Porres -
Christof Ressi -
Dan Wilcox -
IOhannes m zmoelnig -
IOhannes m zmölnig
-
Lucas Cordiviola -
Thomas Grill