so to answer you're question i am leaving the problem open for the moment, but feel free to come up with a nice solution of course ;)
i think the audio playback can be frame-based too, leaving the responsability to the patch to navigate in frames and sound accordingly, like playing sound from frame 578, then 23, etc, ...
audio should be loaded in a buffer and then read in real time with a classic dsp function, that's funny because mp3amp~ works this way already, it receives audio data from the netork and fills in a buffer.
i know the synchronisation would be loose but only well-trained eyes would be able to see any difference.
this would work well on fast machines only.
what do you think ?
best,
sevy/yves