Hallo!
it is true that the gpu is unused most of the time and could be used to do fast calculations, but the main output for graphicscards is the screen and reading back into ram can be slow. (of course "slow" is a relative term). I also see a problem in the different formats for audio and images. most graphics cards are optimized for vec4 processings (4 parallel color channels). and audio...
Well, there are already quite some papers/software for audio processing on the gpu (see e.g. http://www.gpgpu.org/cgi-bin/blosxom.cgi/Audio%20and%20Signal%20Processing/i...).
But AFAIK nobody did this with pd up to now ...
LG Georg