The GL documentation is unclear on which. The official word is here:
http://www.opengl.org/sdk/docs/man/xhtml/glGetError.xml There is not enough memory left to execute the command.
The state of the GL is undefined,
except for the state of the error flags,
after this error is recorded."
You can easily check main memory usage with top or Window's process manager thingy. I have tested OSX extensively enough to say that pix_film/movie are extremely unlikely to leak memory. WIndows is less tested but it should run for days or weeks constantly loading and unloading movies (36 days is the longest I have gotten a WinXP machine to run without something going wrong).
Texture VRAM should be paged out to main memory if the card if full and performance will suffer greatly when this happens. If you have a system with shared CPU and GPU memory then that could be a problem too.
You have not said what OS and hardware you have, which would be very helpful.