Well, Lua really shines for dynamic and generative systems. Try making dynamics graphics with polyphony in Pd and see how far you get. Things I wouldn't do _directly_ in Lua include video processing, DSP, etc. basically things with iterations over lots of data and intensive math ops. That said, I would still do them in Lua, but I would do them via C/C++ bindings that handle all of the really processing intensive work.
My basic approach to Lua is to use it as a sketching interface first and figure out how I want things to work and then once I get a feel for it, move certain sections down to C/C++. When you make this transition, it's best to generalize so that you can reuse it later. I do this for intensive operations like matrix processing, vector-ops, buffer drawing etc. First thing to do is get yourself some good vector math and quaternion bindings. I use these all the time and they save me a ton of work. I can send you a nice quaternion lib if you need it. Also, get yourself some mid-level OpenGL abstractions bound to Lua as userdata like Textures, Shaders, FBOs, etc that handle all of the dirty work under the hood and provide a clean interface to Lua. For example, here's some render-to-texture and post processing code as I typically use it in my own code:
local scene = Texture(1024, 768) local blurred_scene = Texture(1024, 768) local slab = Slab("blur.glsl") slab:param("width", {2, 2})
scene:begincapture(0) //draw scene here scene:endcapture(0)
slab:draw(scene, blurred_scene)
blurred_scene:bind(0) gl.Begin("QUADS") //draw a quad with tex coords gl.End() blurred_scene:unbind(0)
So, using a few higher-level C++ objects, you make make really concise scripts that do quite a bit. As you can see, it's really simple to render to a texture and apply effects to it given a representation of a Texture and a Slab. What's nice is that the slabs and can be arbitrarily chained together for more complicated effects and feedback with delay lines of varying length. The code for doing this stuff should be ready to share come the new year.
Taking this method to the extreme a friend of mine has made a scriptable DSP system using Lua called Vessel which is bundled into the Lua~ max external. The source is available and could easily be built into a pd external. See: http://www.mat.ucsb.edu/%7Ewakefield/lua%7E/lua%7E.htm .
If you're interested in discussing Lua for audiovisual composition, we've established a mailing list on the Media Arts and Technology servers here at UC Santa Barbara called lua-av ( http://zydeco.mat.ucsb.edu/mailman/listinfo/lua-av ). Feel free to sign up. There's very little traffic on it right now, but then again there are only about 5 people on the list. I don't think there are that many people doing audiovisual work with Lua right now anyway, but I anticipate more traffic once we get our standalone Lua-av software released and the utility of Lua for this kind of thing becomes apparent beyond the gaming industry. Most of the discussions about Lua and OpenGL right now are on the Jitter mailing list and it would be nice to consolidate these discussions into a central location that isn't Jitter, GEM, etc. specific.
best, wes