It looks about the same to me though, the only gem functions I'm seeing using a lot of CPU are colour conversion (I guess for the bttv capture) and the rubber dynamics stuff.)
That is not all that helpful. Surely there is a way to get the
real call tree. All of the time spent in libGLcore should be
traced back to calls in GEM. That info is useful in your
situation.
Can you just list the numbers of non-pix GEM objects you are using?