the solution is to use 2 instances of pd: pd -noaudio for gem. pd -rt for audio.
for communication between audio & video i used netsend / netreceive or you can use OSC.
Wow, that sounds like a major irritation. How about increasing the buffer size of the audio? I don't know if that helps or would be at an acceptable level for live work...
Kevin