Hi Ivo,
the combination of pd -rt and gem with kernels up to 2.6.11 is known to cause this behavior. Upgrading the kernel will help (ouch) or disabling -rt when using Gem, or in cases when you need both, lolat audio and 3D graphics, split your patch in two, running on two instances of pd, one for -rt audio, the other one for Gem, and communicate the data via netsend.
Hope this helps,
Peter