Dear list, dear Miller,
I tried Pd on a fresh Raspbian install today. I removed pulseaudio as you suggested. I installed Pd from the Raspbian repos (not your version). I tried it with my USB E-MU 0404 soundcard. It worked right away. No need to slow down the USB. I tried with a latency in Pd of 10 ms, in X, connected to the internet, and there was very little dropouts. This is so promising it's scary.
Now i have one very frustrating problem : i don't know how to choose my soundcard from the command line when starting Pd... It shows up as Card 1, device 0 when i do "aplay -l", but i don't know how to use this information.
Cheers!
Pierre.