hi,
TL;DR i'd like to suggest to use deken-specifiers as (optional) part of external filenames, in order to allow co-installability of externals of different OSs, architectures and floatsizes (and more to come).
i would really love to push the double precision saga towards a (happy) end. we have been able to compile Pd for 64bit double precision numbers. there's even a double-precision variant available in the Debian "experimental" repositories (but who knows that?)
*very* few people have started to provide externals (i counted: 4).
afaict the biggest hurdle is that you can't really co-install single and double variants of the same external. since there are so few double-precision externals available, people who rely on externals will be forced to use single-precision Pd for some time. but since installing a double-precision external might overwrite an existing single-precision external (required in your other project), i understand why people are not exactly eager to do that.¹
one solution to this problem is to use different installation paths (e.g. ~/Documents/Pd/extra/ vs ~/Documents/Pd/extra64/). this doesn't play well with how deken currently works (as it stores the installation path globally (for all versions/variants of Pd).
Lucas suggested to use different file extensions (a year ago...time flies), by inserting `.float64` (and possibly `.float32`) right before our known extension (so we get `foo.float64.m_amd64`) I didn't especially like this back then, but in the meantime i've come to the conclusion that it's probably the best way forward.
however, i think that we might do better than just inserting a single `.float64`, and just unify the entire naming scheme to hold all the information we need.
i'd therefore suggest to use the deken-specifier together with the native extension (for dynamic-link libraries), as a new extension.
the "native extension for dynamic-link libraries" is typically defined on an OS level, and is something like ".dll" on Windows, ".dylib" on macOS and ".so" in the un*x world.
the "deken-specifier" is what we use in deken packages to know that they contain binaries for your specific combination of CPU, OS and precision, and looks like "<OS>-<CPU>-<precision>", e.g. "Darwin-arm64-32" (which denotes a macOS binary ("Darwin") that runs on the M1 processor ("arm64") and uses single-precision numbers ("32" bits).
this would give us filenames like "zexy.windows-amd64-32.dll" to keep things simple (and reduce the noise with -verbose), i would suggest to only allow lower case specifiers, and no arch variants (e.g. i386 for all x86_32 variants, and amd64 for all x86_64 variants)
pros
- using the system extension does not require us to invent our own extension for each new platform - system tools often use the file-extension to recognize the file type - deken-specifiers fully cover what we need to know (the problem space is the same for deken package files and externals: allow coexistence of files with multiple OS/arch/precision specs) - people can relate the files within a deken-package with the deken-package-filename - if we ever need to add a new parameter, the deken specifier and the externals are likely to be affected in a similar way, so we need to solve the problem only once. - it gets rid of the super-cryptic .<first-letter-of-the-os>_<cpu-arch> specifier (.o_ia64 anybody?)
cons
it shares the same (final) extensions as any support libraries. eg. "zexy.linux-amd64-32.so" + "libzexy.linux-amd64-32.so" (or even libzexy.linux-amd64-32.so.so, but I guess we don't want this)
probably some more...
instead of using the system extension for dynamic libraries, we might pick a general unified (final) extension, instead of the system ones, e.g. .pdx (but that is already taken) or .pd_external. but i think the less we invent ourselves, the better.
Lucas had started a feature-request/discussion on this very topic a year ago, but it was dormant until now.
https://github.com/pure-data/pure-data/issues/902
i would like to hear your opinion on this (here or at the issue tracker; or both), and eventually get this done.
once this is solved, i will start to push Pd64 packages to the Debian repositories, so people can start to use it (without having to compile themselves).
gfmsdr IOhannes
¹ just for the record: the biggest hurdle is of course that there is no double-precision download available right now... but that's a bit of an egg-hen problem.
Yeah I guess it makes sense to have a distinct extension.
I haven't provided double precision externals yet because I think the binary should be easily available for those unaware of compiling magic first. And while we're at it, I guess it's time to provide them at miller's site and puredata.info
cheers
Em ter., 29 de mar. de 2022 às 06:29, IOhannes m zmoelnig zmoelnig@iem.at escreveu:
hi,
TL;DR i'd like to suggest to use deken-specifiers as (optional) part of external filenames, in order to allow co-installability of externals of different OSs, architectures and floatsizes (and more to come).
i would really love to push the double precision saga towards a (happy) end. we have been able to compile Pd for 64bit double precision numbers. there's even a double-precision variant available in the Debian "experimental" repositories (but who knows that?)
*very* few people have started to provide externals (i counted: 4).
afaict the biggest hurdle is that you can't really co-install single and double variants of the same external. since there are so few double-precision externals available, people who rely on externals will be forced to use single-precision Pd for some time. but since installing a double-precision external might overwrite an existing single-precision external (required in your other project), i understand why people are not exactly eager to do that.¹
one solution to this problem is to use different installation paths (e.g. ~/Documents/Pd/extra/ vs ~/Documents/Pd/extra64/). this doesn't play well with how deken currently works (as it stores the installation path globally (for all versions/variants of Pd).
Lucas suggested to use different file extensions (a year ago...time flies), by inserting `.float64` (and possibly `.float32`) right before our known extension (so we get `foo.float64.m_amd64`) I didn't especially like this back then, but in the meantime i've come to the conclusion that it's probably the best way forward.
however, i think that we might do better than just inserting a single `.float64`, and just unify the entire naming scheme to hold all the information we need.
i'd therefore suggest to use the deken-specifier together with the native extension (for dynamic-link libraries), as a new extension.
the "native extension for dynamic-link libraries" is typically defined on an OS level, and is something like ".dll" on Windows, ".dylib" on macOS and ".so" in the un*x world.
the "deken-specifier" is what we use in deken packages to know that they contain binaries for your specific combination of CPU, OS and precision, and looks like "<OS>-<CPU>-<precision>", e.g. "Darwin-arm64-32" (which denotes a macOS binary ("Darwin") that runs on the M1 processor ("arm64") and uses single-precision numbers ("32" bits).
this would give us filenames like "zexy.windows-amd64-32.dll" to keep things simple (and reduce the noise with -verbose), i would suggest to only allow lower case specifiers, and no arch variants (e.g. i386 for all x86_32 variants, and amd64 for all x86_64 variants)
pros
- using the system extension does not require us to invent our own
extension for each new platform
- system tools often use the file-extension to recognize the file type
- deken-specifiers fully cover what we need to know (the problem space
is the same for deken package files and externals: allow coexistence of files with multiple OS/arch/precision specs)
- people can relate the files within a deken-package with the
deken-package-filename
- if we ever need to add a new parameter, the deken specifier and the
externals are likely to be affected in a similar way, so we need to solve the problem only once.
- it gets rid of the super-cryptic .<first-letter-of-the-os>_<cpu-arch>
specifier (.o_ia64 anybody?)
cons
it shares the same (final) extensions as any support libraries. eg. "zexy.linux-amd64-32.so" + "libzexy.linux-amd64-32.so" (or even libzexy.linux-amd64-32.so.so, but I guess we don't want this)
probably some more...
instead of using the system extension for dynamic libraries, we might pick a general unified (final) extension, instead of the system ones, e.g. .pdx (but that is already taken) or .pd_external. but i think the less we invent ourselves, the better.
Lucas had started a feature-request/discussion on this very topic a year ago, but it was dormant until now.
https://github.com/pure-data/pure-data/issues/902
i would like to hear your opinion on this (here or at the issue tracker; or both), and eventually get this done.
once this is solved, i will start to push Pd64 packages to the Debian repositories, so people can start to use it (without having to compile themselves).
gfmsdr IOhannes
¹ just for the record: the biggest hurdle is of course that there is no double-precision download available right now... but that's a bit of an egg-hen problem. _______________________________________________ Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
On 3/29/22 16:08, Alexandre Torres Porres wrote:
I haven't provided double precision externals yet because I think the binary should be easily available for those unaware of compiling magic
i'm not exactly sure what you mean here.
1st: you do not need a compiled version of Pd in order to produce a double-precision external (at least on macOS and Linux). so what are you waiting for?
2nd: i would rather have a handful of developers jump through the hoop of compiling a Pd-double for themselves in order to test their externals, than all Pd users on the earth download a Pd-double for which there are no externals available.
as a developer (of compiled externals) i hope you can be asked to build a Pd. the INSTALL.txt is quite detailed. or put otherwise: should i trust a developer who is not able to build Pd?
gfmrd IOhannes
PS: it's a bit disappointing though, that the first comment in this discussion already digresses :-P
Em ter., 29 de mar. de 2022 às 12:10, IOhannes m zmoelnig zmoelnig@iem.at escreveu:
i'm not exactly sure what you mean here.
1st: you do not need a compiled version of Pd in order to produce a double-precision external (at least on macOS and Linux). so what are you waiting for?
I just don't think it makes sense to stop my workflow and concentrate on working on this if most people aren't using it yet, once the binary isn't available. So yeah, I'm waiting for the binary to come up for everyone. But I get your point that "all externals" should be made available. I don't care about "all" and I can deal with my stuff, which I hope to be able to take care of quite quickly once the time is right. Whenever the binary is out there I can provide ELSE and Cyclone in a week or two the latest (I hope).
But I don't really care, if we're organizing a work force to upload "all" externals first, let's do it then, I will follow...
as a developer (of compiled externals) i hope you can be asked to build a Pd. the INSTALL.txt is quite detailed. or put otherwise: should i trust a developer who is not able to build Pd?
I'm able to build Pd alright, but I don't think people should count on that to trust me as a developer :) (I'm always open to say "don't trust me" anyway, haha). But then, my point is, again, most people don't build Pd...
And here's another consideration. Some people (I know it's just a few people) don't use externals and would be happy to try patches and start working on this even though "all" externals aren't out there yet.
gfmrd IOhannes
PS: it's a bit disappointing though, that the first comment in this discussion already digresses :-P _______________________________________________ Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
+1
I think it's nicer to use a common extension and have the platform/arch/floatsize specifier as a seperate component.
On 29.03.2022 11:28, IOhannes m zmoelnig wrote:
hi,
TL;DR i'd like to suggest to use deken-specifiers as (optional) part of external filenames, in order to allow co-installability of externals of different OSs, architectures and floatsizes (and more to come).
i would really love to push the double precision saga towards a (happy) end. we have been able to compile Pd for 64bit double precision numbers. there's even a double-precision variant available in the Debian "experimental" repositories (but who knows that?)
*very* few people have started to provide externals (i counted: 4).
afaict the biggest hurdle is that you can't really co-install single and double variants of the same external. since there are so few double-precision externals available, people who rely on externals will be forced to use single-precision Pd for some time. but since installing a double-precision external might overwrite an existing single-precision external (required in your other project), i understand why people are not exactly eager to do that.¹
one solution to this problem is to use different installation paths (e.g. ~/Documents/Pd/extra/ vs ~/Documents/Pd/extra64/). this doesn't play well with how deken currently works (as it stores the installation path globally (for all versions/variants of Pd).
Lucas suggested to use different file extensions (a year ago...time flies), by inserting `.float64` (and possibly `.float32`) right before our known extension (so we get `foo.float64.m_amd64`) I didn't especially like this back then, but in the meantime i've come to the conclusion that it's probably the best way forward.
however, i think that we might do better than just inserting a single `.float64`, and just unify the entire naming scheme to hold all the information we need.
i'd therefore suggest to use the deken-specifier together with the native extension (for dynamic-link libraries), as a new extension.
the "native extension for dynamic-link libraries" is typically defined on an OS level, and is something like ".dll" on Windows, ".dylib" on macOS and ".so" in the un*x world.
the "deken-specifier" is what we use in deken packages to know that they contain binaries for your specific combination of CPU, OS and precision, and looks like "<OS>-<CPU>-<precision>", e.g. "Darwin-arm64-32" (which denotes a macOS binary ("Darwin") that runs on the M1 processor ("arm64") and uses single-precision numbers ("32" bits).
this would give us filenames like "zexy.windows-amd64-32.dll" to keep things simple (and reduce the noise with -verbose), i would suggest to only allow lower case specifiers, and no arch variants (e.g. i386 for all x86_32 variants, and amd64 for all x86_64 variants)
pros
- using the system extension does not require us to invent our own
extension for each new platform
- system tools often use the file-extension to recognize the file type
- deken-specifiers fully cover what we need to know (the problem space
is the same for deken package files and externals: allow coexistence of files with multiple OS/arch/precision specs)
- people can relate the files within a deken-package with the
deken-package-filename
- if we ever need to add a new parameter, the deken specifier and the
externals are likely to be affected in a similar way, so we need to solve the problem only once.
- it gets rid of the super-cryptic
.<first-letter-of-the-os>_<cpu-arch> specifier (.o_ia64 anybody?)
cons
it shares the same (final) extensions as any support libraries. eg. "zexy.linux-amd64-32.so" + "libzexy.linux-amd64-32.so" (or even libzexy.linux-amd64-32.so.so, but I guess we don't want this)
probably some more...
instead of using the system extension for dynamic libraries, we might pick a general unified (final) extension, instead of the system ones, e.g. .pdx (but that is already taken) or .pd_external. but i think the less we invent ourselves, the better.
Lucas had started a feature-request/discussion on this very topic a year ago, but it was dormant until now.
https://github.com/pure-data/pure-data/issues/902
i would like to hear your opinion on this (here or at the issue tracker; or both), and eventually get this done.
once this is solved, i will start to push Pd64 packages to the Debian repositories, so people can start to use it (without having to compile themselves).
gfmsdr IOhannes
¹ just for the record: the biggest hurdle is of course that there is no double-precision download available right now... but that's a bit of an egg-hen problem.
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
On Tue, 2022-03-29 at 17:29 +0200, Christof Ressi wrote:
+1
+1
I think it's nicer to use a common extension and have the platform/arch/floatsize specifier as a seperate component.
I didn't especially like this back then, but in the meantime i've come to the conclusion that it's probably the best way forward.
Why? I think it is much friendlier for the user to see in the filename what is in it. If binaries are distinguished by installing them to separated folders (but still share filename), people will try to move files around to make things work and thus getting into a mess really quickly. One shouldn't have to use 'file pdexternal.ext' to know what actually is in it.
Having said that, I'm still curious to know what you thought are the cons back then. Roman
One "con" I can imagine is that there are "cleaner" alternatives than name mangling.
For example, VST3 plugins use a bundle structure which also allows for "merged bundles": https://developer.steinberg.help/display/VST/Plug-in+Format+Structure.
Something similar *could* work for Pd externals:
foo-lib/
-- bar.pd/
---- windows-amd64-64/bar.dll
---- windows-amd64-32/bar.dll
---- linux-amd64-64/bar.dll
---- linux-amd64-32/bar.dll
etc.
In this case, the .pd extension would be used both for Pd abstractions and external binary bundles.
But it might be overkill...
Christof
On 29.03.2022 17:49, Roman Haefeli wrote:
On Tue, 2022-03-29 at 17:29 +0200, Christof Ressi wrote:
+1
+1
I think it's nicer to use a common extension and have the platform/arch/floatsize specifier as a seperate component.
I didn't especially like this back then, but in the meantime i've come to the conclusion that it's probably the best way forward.
Why? I think it is much friendlier for the user to see in the filename what is in it. If binaries are distinguished by installing them to separated folders (but still share filename), people will try to move files around to make things work and thus getting into a mess really quickly. One shouldn't have to use 'file pdexternal.ext' to know what actually is in it.
Having said that, I'm still curious to know what you thought are the cons back then. Roman
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
Joining the discussion:
I think the "deken-specifier" is Ok.
We should probably be prepared for both apps installations coexisting. In the case of Windows we should have a different default location and a name for the double app. Think of Pd-double whose default installation via the installer lives next to the single precision app:
C:\Program Files\Pd
C:\Program Files\Pd-double.
"Pd-double" can be anything better or whatever.
I think we can live sharing the same "Windows registry settings" for both apps.
We need the different app name not only for Windows. Thinking of the debian package and of course the macos app.
--
Mensaje telepatico asistido por maquinas.
On 3/29/22 20:03, Lucas Cordiviola wrote:
Joining the discussion:
I think the "deken-specifier" is Ok.
here's something i just came up with: what would the deken-specifier be for fat-binaries (on macOS).
i do not want to have zexy.darwin-amd64-32.darwin-arm64-32.so (the reason is practicability: Pd constructs the filename beforehand, and then tries to open the file. but in this example, we do not really know whether the filename is "zexy.darwin-amd64-32.darwin-arm64-32.so" or "zexy.darwin-arm64-32.darwin-amd64-32.so" or "zexy.darwin-arm64-32.darwin-ppc-32.darwin-amd64-32.so")
we obviously could ditch fat binaries in this case altogether (not sure whether this is a good idea though).
gfmas IOhannes
i do not want to have zexy.darwin-amd64-32.darwin-arm64-32.so
maybe a bundle structure (as described in https://lists.puredata.info/pipermail/pd-dev/2022-03/022997.html) might not be such a bad idea after all?
On 3/30/22 16:16, Christof Ressi wrote:
i do not want to have zexy.darwin-amd64-32.darwin-arm64-32.so
maybe a bundle structure (as described in https://lists.puredata.info/pipermail/pd-dev/2022-03/022997.html) might not be such a bad idea after all?
maybe. it solves problems like auxiliary libraries and keeps the directory reasonably clean.
but i don't really see how it would help with fat binaries.
dgfmsfa IOhannes
but i don't really see how it would help with fat binaries.
Two solutions that come to my mind:
1) just use an ugly folder name:
foo.pd/darwin-amd64-32.darwin-arm64-32/foo.dylib
Typically, the user won't see it :-)
2) use a special specifier for universal binaries:
foo.pd/darwin-universal-32/foo.dylib
On 30.03.2022 16:33, IOhannes m zmoelnig wrote:
On 3/30/22 16:16, Christof Ressi wrote:
i do not want to have zexy.darwin-amd64-32.darwin-arm64-32.so
maybe a bundle structure (as described in https://lists.puredata.info/pipermail/pd-dev/2022-03/022997.html) might not be such a bad idea after all?
maybe. it solves problems like auxiliary libraries and keeps the directory reasonably clean.
but i don't really see how it would help with fat binaries.
dgfmsfa IOhannes
Pd-dev mailing list Pd-dev@lists.iem.at https://lists.puredata.info/listinfo/pd-dev
On 3/30/22 17:21, Christof Ressi wrote:
but i don't really see how it would help with fat binaries.
Two solutions that come to my mind:
- just use an ugly folder name:
foo.pd/darwin-amd64-32.darwin-arm64-32/foo.dylib
the problem with this is, that it is not well-defined.
currently we calculate a few variants of possible names based on a reference string ("darwin-arm64-32") and try to open files according to these names.
but we cannot possible foresee all the possible variants; even if we limit ourselves to 4 architectures, we get about 40 possible variants (for any 1 architecture)
so we must do this differently: glob all files (or directories) and see whether one of the results matches our reference string.
i'm not sure i want this: there seems to be much too much logic involved for little apparent gain.
Typically, the user won't see it :-)
the user will notice if loading libraries takes forever.
dgfamdsr IOhannes
On 30.03.2022 17:40, IOhannes m zmoelnig wrote:
On 3/30/22 17:21, Christof Ressi wrote:
but i don't really see how it would help with fat binaries.
Two solutions that come to my mind:
- just use an ugly folder name:
foo.pd/darwin-amd64-32.darwin-arm64-32/foo.dylib
the problem with this is, that it is not well-defined.
currently we calculate a few variants of possible names based on a reference string ("darwin-arm64-32") and try to open files according to these names.
but we cannot possible foresee all the possible variants; even if we limit ourselves to 4 architectures, we get about 40 possible variants (for any 1 architecture)
I see. What about the second idea?
- use a special specifier for universal binaries:
foo.pd/darwin-universal-32/foo.dylib
-
Alexandre Torres Porres -
Christof Ressi -
IOhannes m zmoelnig -
Lucas Cordiviola -
Roman Haefeli