A few notes about -rt mode:
First, -rt is different on OSX compared to Linux. On Linux, the flag sets the priority of the pd process just like nice or renice, but on OSX -rt set priority using a pthread scheduling call. The difference is due to how each OS handles scheduling of tasks - Linux by PID and OSX by threads. The proper way to schedule a 'real-time' thread on OSX is to use the kernel threading API which guarantees time slices for a given process. Pd would have to undergo a but of reworking to use this though (the DSP would have to go in it's own thread separate from the GUI and main routines).
Second, the issues with -rt and video/GL appear to be different for each platform. On OSX, the problem is that the pthread scheduling works too well. GEM simply locks out all other UI processes if CPU load gets too high. This may appear like a freeze or lockup, but the system is actually chugging along as if everything is fine. GEM performance with the higher priority enabled is phenomenal - no dropped frames, immediate movie loading, and low latency. It's just far too easy to have GEM lock you out of all other UI though, which is not a good tradeoff.
Following Tom's idea about turning RT on and off by routine, I should look into something like setting the pthread priority right before and immediately after the render callbacks. This might give the render chain enough time to work while providing a breather for the task manager to allocate time to other lower priority processes. I have no idea what the overhead would be for calling these many times a second.
cgc