the solution is to use 2 instances of pd:
pd -noaudio for gem.
pd -rt for audio.

for communication between audio & video i used netsend / netreceive or you
can use OSC.

Wow, that sounds like a major irritation. How about increasing the buffer size of the audio? I don't know if that helps or would be at an acceptable level for live work...

Kevin