The issue is with the h.264 codec. On the Mac, compress them as 'Apple Intermediate Codec' or ProRes (which comes with what's left of Final Cut 'Pro'). The files will be much larger in size on the drive but play back much better. When I wrote the OSX pix_film/movie code long ago, it was only intended to play back intraframe codecs like the JPEG based ones and not MPEG which are consumer delivery formats.
You should also set the gemwin to render at least 30 frames per second and for smoothest playback use 60fps which is the refresh rate of an LCD. I think the default is still 15 or 20fps?