If you haven't tried it already, try increasing the latency in the audio settings (probably not the issue, but worth a shot).
Maybe it's only having trouble with duplex audio (simultaneous input and output). Do you have both input and output enabled for your soundcard? Does it work if you only have the output device set, and leave the input device blank?
You can test to see if duplex audio is working outside of pd - try running "arecord | aplay" (the vertical bar means pipe the data from arecord into aplay), and see if you get usable sound. You may need to set some command line flags on those to get the sample rate and audio devices right, sorry I don't know them offhand.
I am running a fairly similar setup - for an honors project at my university, I'm using a Raspberry Pi and Behringer UCG102. I have a nice multi-effects pedal working at less than 20ms latency. If my Raspberry Pi can handle it, I feel like your UDOO quad should definitely be able to. I'd like to add that you have a nice taste in projects :)