Hello,
Finally my double-precision-Pd efforts resulted in code decent enough to be useful in practice. It's all documented on:
http://www.katjaas.nl/doubleprecision/doubleprecision.html
From there you can download a .zip with two .patch files to make vanilla Pd
0.43-0 double precision ready. In fact what you have is 'arbitrary-precision' code which can be built for single or double precision with the setting of a single definition in the API header m_pd.h. Test patches are included.
So far, I have tested on OSX and Linux. Remarkably, double precision Pd performance is comparable to current Pd on the Intel CPU's. The only drawback that I can think of is, it eats memory. This would be a bummer for Pd users doing Ableton style stuff with lots of soundfiles to be loaded into RAM. Krzysztof Czaja already warned me for this at Pd Con, but at the time I forgot a bit about the problems with loading soundfiles during a live session.
For me, as a precision freak, it is a delight to work with double precision Pd. Some examples are illustrated on mentioned webpage. Unfortunately, my music stuff needs Pd extended. I would be happy to continue the project.
Katja
Hey Katja,
This is great, just starting to dig into it. Its a great write-up too. I'd like to put together some Pd-extended nightly builds based on this, and start working out a work flow for all the fixes that we will inevitably need to do. To start with, I think you should request SVN commit access so you can directly commit fixes to externals as we find them. I know there will be loads of changing float to t_float and things like that, and I imagine a few places that'll need more work.
https://puredata.info/docs/developer/SVNCommitAccess
Then next, I think it makes sense to start a git fork to work on this. github.com is a pretty easy place to put it, or gitorious.org. Then I can setup a nightly build based on that repo.
I just looking thru your patches, a couple of quick comments to make the patches a lot more readable. Basically, before submitting, read the output of 'git diff' and try to make it so that the only changes that appear are ones that change the function of the code, not how the code looks. That makes for a shorter and much more readable patch.
* keep the indentation the same, in a few places in the patch, it seems that the only difference is the indentation. It should be all spaces, no tabs, with 4 spaces as a single level of indentation.
* try to eliminate spacing changes like: - -#ifndef N32 +#ifndef N32 t_float qsqrt(t_float f) {return (q8_sqrt(f)); } t_float qrsqrt(t_float f) {return (q8_rsqrt(f)); } #endif
- - typedef struct sigrsqrt
* keep variable names the same, when it makes sense, for example: - x->x_arrayname = s; + x->arrayname = s;
- float x_f; /* scalar frequency */ + t_float f; // scalar frequency
* it seems you define this union a lot in d_math.c, why not just once at the top? + union + { + float f; + int32_t l; + }u;
* same with GOODINT, perhaps that should just be defined once in a header?
* also, I think the %g printf pattern should handle the number of decimals, so perhaps in print_float it should just be %g or %lg.
.hc
On Sat, 2011-10-01 at 03:12 +0200, katja wrote:
Hello,
Finally my double-precision-Pd efforts resulted in code decent enough to be useful in practice. It's all documented on:
http://www.katjaas.nl/doubleprecision/doubleprecision.html
From there you can download a .zip with two .patch files to make vanilla Pd 0.43-0 double precision ready. In fact what you have is 'arbitrary-precision' code which can be built for single or double precision with the setting of a single definition in the API header m_pd.h. Test patches are included.
So far, I have tested on OSX and Linux. Remarkably, double precision Pd performance is comparable to current Pd on the Intel CPU's. The only drawback that I can think of is, it eats memory. This would be a bummer for Pd users doing Ableton style stuff with lots of soundfiles to be loaded into RAM. Krzysztof Czaja already warned me for this at Pd Con, but at the time I forgot a bit about the problems with loading soundfiles during a live session.
For me, as a precision freak, it is a delight to work with double precision Pd. Some examples are illustrated on mentioned webpage. Unfortunately, my music stuff needs Pd extended. I would be happy to continue the project.
Katja _______________________________________________ Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
Hi Hans,
Thanks for your detailed comments. I will go through the code once again, you're right it's not as clean as could be.
Regarding your suggestion to set up a repo, it seems to be the most logical thing to do. This could be considered a temporary branch for the double precision thing, to be discontinued once all the code works fine for single and double precision alike and merged into Pd. I'll start that up one of these days (my first repo ever, hope to get it done...).
A few words on the scope of tests done so far. Rewritten code was developed and tested on Intel core CPU's, where it seems to work smooth, both with SSE and FPU instructions. Regarding functionality and robustness, I have good confidence that it will work anywhere, as it's simple and depends only on standard libs. But considering performance, more checking is needed. The rewritten phase-wrapping classes have branches in the perform-loops. These are only executed in rare cases, therefore branch prediction mechanisms can do their good work. But on older architectures branching may be more expensive. That was also the reason for Miller's branchless design of these classes, of course. PPC in particular should be considered, as there are still quite a few users. Maybe the code needs some finetuning to PPC. Only after settling this aspect, I would consider submitting the double-precision-ready .patch file for Pd's core code. Otherwise I'd risk an endless series of amendements on a submitted patch. In the meantime, double precision ready code would be available from this git repo we're planning.
Katja
Sounds great. I'm happy to help setup a git repo if you want me too. github and gitorious are pretty straightforward to get the initial repo, then it would be a matter of pushing the pure-data.git to that repo, and starting work from there. I think it makes sense to work off of pure-data.git rather than pd-extended.git since this is a patch targetted at getting into Miller's repo.
As for arch issues, I think Intel and ARM are the big ones to test these days. But PPC is fine too.
.hc
On Oct 2, 2011, at 5:24 PM, katja wrote:
Hi Hans,
Thanks for your detailed comments. I will go through the code once again, you're right it's not as clean as could be.
Regarding your suggestion to set up a repo, it seems to be the most logical thing to do. This could be considered a temporary branch for the double precision thing, to be discontinued once all the code works fine for single and double precision alike and merged into Pd. I'll start that up one of these days (my first repo ever, hope to get it done...).
A few words on the scope of tests done so far. Rewritten code was developed and tested on Intel core CPU's, where it seems to work smooth, both with SSE and FPU instructions. Regarding functionality and robustness, I have good confidence that it will work anywhere, as it's simple and depends only on standard libs. But considering performance, more checking is needed. The rewritten phase-wrapping classes have branches in the perform-loops. These are only executed in rare cases, therefore branch prediction mechanisms can do their good work. But on older architectures branching may be more expensive. That was also the reason for Miller's branchless design of these classes, of course. PPC in particular should be considered, as there are still quite a few users. Maybe the code needs some finetuning to PPC. Only after settling this aspect, I would consider submitting the double-precision-ready .patch file for Pd's core code. Otherwise I'd risk an endless series of amendements on a submitted patch. In the meantime, double precision ready code would be available from this git repo we're planning.
Katja
Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
----------------------------------------------------------------------------
"Making boring techno music is really easy with modern tools, but with live coding, boring techno is much harder." - Chris McCormick
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2011-10-01 17:25, Hans-Christoph Steiner wrote:
Hey Katja,
This is great, just starting to dig into it. Its a great write-up too.
congrats from my side as well.
however, i got 2 remarks regarding the write-up: - - "The situation is similar to Pd for 64 bit Linux, which can not be mixed with externals for 32 bit Linux." is not really true - at least if by "the situation" you mean that you have to take care not to accidentally load a library of the wrong architecture. the dynamic loader will plainly refuse to load a 32bit external on a 64bit machine (and vice versa), which resolves a lot of headaches, as you never experience mysterious crashes because you got the architecture wrong (which is the only problem with getting double precision into Pd)
- - "there is good reason to build double precision Pd for 64 bit systems and single precision Pd for 32 bit systems." i cannot quite follow the argumentation here. of course, in 64bit world, you can address more memory and thus you might not run out of memory as soon as on 32bit systems. i'm a bit concerned about such a proposal in the "symbol collision" part, as it might seem to be a remedy against the symbol collision problem, which it simply is not. we have had 64bit single-precision versions of Pd around for quite some time (just install it in Debian, Ubuntu,...), so we do already have a compat problem here. it would be different, if 64bit Pd would not be there, as then we could just start on fresh grounds.
anyhow, i think double precision Pd is really one of the biggest things since quite some time.
fgmasdr IOhannes
Thanks IOhannes for all your comments and suggestions.
I just realized that there are several ways in which identical symbols for different function definitions could cause a problem and I did not distinguish them.
1. Pd looks for a setup symbol when trying to load an external binary. 2. A loaded external calls an exported function in Pd. 3. Pd calls an exported function in Pd
Case 2. and 3. can only lead to symbol collision when a single precision and double precision Pd are running simultaneously. So far, I have not seen symbol collision happen though I've often ran them simultaneously. I understand that theoretically it's not guaranteed that it won't happen, that's also the reason why it is generally recommended to only export truly global symbols. However I think it is not really a concern, as there is normally no reason to run single and double precision Pd together, apart from testing purposes.
For case 1., protection is needed indeed. As IOhannes' list of possible approaches indicates, it's not a trivial intervention. I've also been thinking of a mechanism where Pd 'probes' a class at load time in order to find it's float type before instantiating any object. Rather it creates a private test instance for that purpose and tries to solicit output and check the size. To program this is not trivial either, if possible at all, but the advantage would be that it does not have consequence for class code.
Actually I think we have time to find and implement a solution because during the double precision development and test period we do not depend on it. If only we find a good way to point different Pd installations to subdirs in their own 'extra' for loading externals. What I meant to say in my previous mail is, that external executables like bonk~ are found from the proper location because Pd apparently knows they are in it's own extra dir, wherever the installation may be located. I do not know where this path is set but we need an option to add more dirs to that local path without using preferences.
Katja
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2011-10-03 17:44, katja wrote:
Thanks IOhannes for all your comments and suggestions.
I just realized that there are several ways in which identical symbols for different function definitions could cause a problem and I did not distinguish them.
- Pd looks for a setup symbol when trying to load an external binary.
- A loaded external calls an exported function in Pd.
- Pd calls an exported function in Pd
Case 2. and 3. can only lead to symbol collision when a single precision and double precision Pd are running simultaneously. So far,
how is that going to happen? by "running simultaneously", do you mean something like this? $ pd32 & $ pd64
i think all modern OSs will protect the memory (including loaded libraries) of an application from other applications. e.g. if i happen to have a library with an exported symbol "sqrt" which viciously returns the (x+1) rather than sqrt(x), and i start an application that links to this library (thus making use of the "bad" sqrt()), this will not magically make Excel forget it's math.
the problem might obivously appear, if Pd would actually create a libpd.so providing all the exported functions, and pd32 / pd64 would make heavy use of those. in which case, pd32 might get a double-precision libpd.so, and thus be not single precision any more.
but this is really not a problem of running the single and double precision on the same machine, but installing them on the same machine.
For case 1., protection is needed indeed. As IOhannes' list of possible approaches indicates, it's not a trivial intervention. I've also been thinking of a mechanism where Pd 'probes' a class at load time in order to find it's float type before instantiating any object. Rather it creates a private test instance for that purpose and tries to solicit output and check the size. To program this is not trivial either, if possible at all, but the advantage would be that it does not have consequence for class code.
i cannot really think of a way to do that. if we only consider signals, we could do some tests (as the object has a well defined interface to acquire and output "numbers"), though i fail to see how we could validate these tests, without knowing what the object is actually supposed to do. if we consider message objects as well, i don't even know how to "properly" send a message that might reveal something useful.
knows they are in it's own extra dir, wherever the installation may be located. I do not know where this path is set but we need an option to add more dirs to that local path without using preferences.
i don't see how this would help. whether those paths can be modified via preferences or only via startup flags, doesn't really matter. if we want them to not be editable at all, i don't see the point in adding them.
people do use the preferences to add paths to find their libraries. if those paths contain libraries expecting the wrong precision we have a problem.
fmasdr IOhannes
I think for now, we'll just have Pd-extended-like monolithic builds which will be easy to use on their own and will include enough libraries to be useful. They can be run standalone, and if need be, we can disable things like ~/pd-externals quite easily.
These kinds of deployment issues really need a lot of real world data to be correctly solved, so I think its not worth going to deep into it until we get some builds out there for people to use.
.hc
On Oct 3, 2011, at 11:44 AM, katja wrote:
Thanks IOhannes for all your comments and suggestions.
I just realized that there are several ways in which identical symbols for different function definitions could cause a problem and I did not distinguish them.
- Pd looks for a setup symbol when trying to load an external binary.
- A loaded external calls an exported function in Pd.
- Pd calls an exported function in Pd
Case 2. and 3. can only lead to symbol collision when a single precision and double precision Pd are running simultaneously. So far, I have not seen symbol collision happen though I've often ran them simultaneously. I understand that theoretically it's not guaranteed that it won't happen, that's also the reason why it is generally recommended to only export truly global symbols. However I think it is not really a concern, as there is normally no reason to run single and double precision Pd together, apart from testing purposes.
For case 1., protection is needed indeed. As IOhannes' list of possible approaches indicates, it's not a trivial intervention. I've also been thinking of a mechanism where Pd 'probes' a class at load time in order to find it's float type before instantiating any object. Rather it creates a private test instance for that purpose and tries to solicit output and check the size. To program this is not trivial either, if possible at all, but the advantage would be that it does not have consequence for class code.
Actually I think we have time to find and implement a solution because during the double precision development and test period we do not depend on it. If only we find a good way to point different Pd installations to subdirs in their own 'extra' for loading externals. What I meant to say in my previous mail is, that external executables like bonk~ are found from the proper location because Pd apparently knows they are in it's own extra dir, wherever the installation may be located. I do not know where this path is set but we need an option to add more dirs to that local path without using preferences.
Katja
Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
----------------------------------------------------------------------------
kill your television
More on actually trying the patch. I tried to apply it to the HEAD of pure-data.git, and one section failed:
pd@debian-lenny-i386 src $ patch -p1 < ../../pd_doubleready/ make_Pd_core_0430_double_ready.patch patching file d_array.c patching file d_math.c patching file d_misc.c Hunk #2 FAILED at 37. 1 out of 2 hunks FAILED -- saving rejects to file d_misc.c.rej
The patch to 'extra' succeeded.
.hc
On Oct 3, 2011, at 11:44 AM, katja wrote:
Thanks IOhannes for all your comments and suggestions.
I just realized that there are several ways in which identical symbols for different function definitions could cause a problem and I did not distinguish them.
- Pd looks for a setup symbol when trying to load an external binary.
- A loaded external calls an exported function in Pd.
- Pd calls an exported function in Pd
Case 2. and 3. can only lead to symbol collision when a single precision and double precision Pd are running simultaneously. So far, I have not seen symbol collision happen though I've often ran them simultaneously. I understand that theoretically it's not guaranteed that it won't happen, that's also the reason why it is generally recommended to only export truly global symbols. However I think it is not really a concern, as there is normally no reason to run single and double precision Pd together, apart from testing purposes.
For case 1., protection is needed indeed. As IOhannes' list of possible approaches indicates, it's not a trivial intervention. I've also been thinking of a mechanism where Pd 'probes' a class at load time in order to find it's float type before instantiating any object. Rather it creates a private test instance for that purpose and tries to solicit output and check the size. To program this is not trivial either, if possible at all, but the advantage would be that it does not have consequence for class code.
Actually I think we have time to find and implement a solution because during the double precision development and test period we do not depend on it. If only we find a good way to point different Pd installations to subdirs in their own 'extra' for loading externals. What I meant to say in my previous mail is, that external executables like bonk~ are found from the proper location because Pd apparently knows they are in it's own extra dir, wherever the installation may be located. I do not know where this path is set but we need an option to add more dirs to that local path without using preferences.
Katja
Pd-dev mailing list Pd-dev@iem.at http://lists.puredata.info/listinfo/pd-dev
----------------------------------------------------------------------------
Mistrust authority - promote decentralization. - the hacker ethic
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2011-10-03 18:32, Hans-Christoph Steiner wrote:
More on actually trying the patch. I tried to apply it to the HEAD of pure-data.git, and one section failed:
pd@debian-lenny-i386 src $ patch -p1 < ../../pd_doubleready/make_Pd_core_0430_double_ready.patch patching file d_array.c patching file d_math.c patching file d_misc.c Hunk #2 FAILED at 37. 1 out of 2 hunks FAILED -- saving rejects to file d_misc.c.rej
this one is quite easy to fix if you inspect the patch manually. anyhow, i patched the sources and pushed them to my github repository, into the "double" branch. https://github.com/umlaeute/pd-vanilla/tree/double
fgmsdr IOhannes
-
Hans-Christoph Steiner -
IOhannes m zmoelnig -
katja