Hi,
Is anyone doing much in the way of GEM development at the moment?
I've been digging into GEM to write a couple of my own objects and fix a few bugs that have reared their heads...
Things that I'm interested in at the moment
- multitexturing (in the GEM todo file)
- render to texture - render to pixbuffer (in the GEM todo file)
- pixel shader programming to do pix_ style effects in compatible video hardware (altho I believe this is vendor specific at the moment).
I've also made some fairly significant changes to the gempixdualobject class to fix a few bugs. As far as I can tell, the class is/was fairly problematic both in operation and during shutdown due to the way the right input behaves.
I've fixed the problems that I've encountered but they're quite drastic. I made the stoprender call part of the daglink structure rather than a method of gembase. This way it can be called on a per-input rather than per-object basis. This fixes all of the shutdown/object deletion crashes that I was having with the released version of the code (0.87)
Has there been any discussion about the design of n-input GEM objects?
daniel
On Thu, 13 Jun 2002, Daniel Heckenberg wrote:
Hi,
Is anyone doing much in the way of GEM development at the moment?
I've been digging into GEM to write a couple of my own objects and fix a few bugs that have reared their heads...
Things that I'm interested in at the moment
multitexturing (in the GEM todo file)
render to texture
render to pixbuffer (in the GEM todo file)
pixel shader programming to do pix_ style effects in compatible video
hardware (altho I believe this is vendor specific at the moment).
I've also made some fairly significant changes to the gempixdualobject class to fix a few bugs. As far as I can tell, the class is/was fairly problematic both in operation and during shutdown due to the way the right input behaves.
hi,
sounds great, I have lately been trying to improve pix_ operations performance with MMX and switching to integer only calculations. .. One thing I ended up doing is to make the imageStruct a real class, which should make the handling of pixes a lot easier.
I've fixed the problems that I've encountered but they're quite drastic. I made the stoprender call part of the daglink structure rather than a method of gembase. This way it can be called on a per-input rather than per-object basis. This fixes all of the shutdown/object deletion crashes that I was having with the released version of the code (0.87)
I would like to have a look at that. I don't really understand what a per input stoprender means.
Has there been any discussion about the design of n-input GEM objects?
You mean n images input ? Not that I can remember.
I have already asked Johannes about setting up a CVS for GEM, this would make shared development a lot easier.
Can you send a diff against gem-0.87 of the changes you did ?
Guenter
From: g|nter geiger hi,
sounds great, I have lately been trying to improve pix_ operations performance with MMX and switching to integer only calculations. .. One thing I ended up doing is to make the imageStruct a real class, which should make the handling of pixes a lot easier.
That's good news: those pix_es could definitely be optimized and MMX and Altivec are great for image processing. How far are you through this process?
I've fixed the problems that I've encountered but they're quite
drastic. I
made the stoprender call part of the daglink structure rather
than a method
of gembase. This way it can be called on a per-input rather
than per-object
basis. This fixes all of the shutdown/object deletion crashes
that I was
having with the released version of the code (0.87)
I would like to have a look at that. I don't really understand what a per input stoprender means.
There are problems in the release code when an dualpix object is deleted or its stoprender method is called. When stoprender is called (by traversing down either gemlist) both of its cache pointers are NULLed. This means that when the object is deleted it can't call breakDAG for the other gemlist. If the objects are deleted in the "wrong" order when rendering is occuring (say by closing the patch) then this can cause a crash.
To solve this I added a stoprender function to the render and postrender functions that are kept in the DAG list. Now a per-input stoprender function can simply clear that input's cache and if/when the object is deleted the unbroken DAGs can be properly broken. There may well be a better/more elegant way to do this, but this seems to fit with the present code.
The other obvious alternative is to pass the message down the PD connections (as happens when the DAG is constructed).
There's another problem in the GemPixDualObj class, in its RightRender method. GemPixDualObj copies the GemState->image pointer to use later when its left render occurs.
This fails in the following case: If there are pix_fx objects in this right gemlist, they will reset the GemState->image pointer back to their "oldImage" values when their postrender functions are called. That means that when the GemPixDualObj Render occurs, its m_pixRight member will not be pointing to the correct image. This can cause strange problems like the [pix_data] problem that I reported, as well as more catastrophic problems if the image format is converted upstream (ie if there is a [pix_rgba] in the fx chain that is effectively rolled back before the GemPixDualObj occurs.)
As the pointer copy occurs regardless of the newImage flag, another problem arises: If pix_fx objects are in a chain and going into the right input of a GemPixDualObj and the render chain is running without the newImag flag set, the fx won't run but the image will be taken anyway.
Has there been any discussion about the design of n-input GEM objects?
You mean n images input ? Not that I can remember.
Looking through the per-input code above, the obvious question is "how to make this easily extend to n-input objects?". The motivation for n-input objects seems pretty clear: e.g doing n layer compositing would be much neater and faster that way.
I have already asked Johannes about setting up a CVS for GEM, this would make shared development a lot easier.
Yes, that would be good. I note that there's already a CVS archive for the GEM OSX project so it would be good to get a central repository up to keep things consistent and organised.
Can you send a diff against gem-0.87 of the changes you did ?
Sure, see attached.
I'll provide the following notes: GemBase.h/cpp: breakDAG/stoprender system changed to provide per-input stoprender calls GemPixDualObj.h/cpp: breakDAG/stoprender system changed to provide per-input stoprender calls. right render method now takes a copy of the current pix_block pix_filmNT.h/cpp: added error handling to film opening to handle codec/decompression issues hacked in support to load 8-bit palettized AVIs and output them as greyscale.
I've also written a fairly trivial pix_snap2tex object that snaps the frame buffer to a texture (good for diffusion/feedback/motion blur effects).
On Fri, 14 Jun 2002, Daniel Heckenberg wrote:
That's good news: those pix_es could definitely be optimized and MMX and Altivec are great for image processing. How far are you through this process?
Thats a good questions, I have only started. So far I have done the simple things like pix_add, pix_sub which got a tremendous speedup.
I have been working with pix_convolve a bit, no asm yet, but by specializing to fixed convolution kernels and avoiding allocation/deallocation of memory as well as floating point calculations I got a 2-3 fold improvement in speed. The asm code is a bit more complex there and I am not a specialist in doing this.
Finally we have to check the pix_ objects step by step,and of course non asm version have to be supplied to .... maybe investigate on libraries for doing that.
There are problems in the release code when an dualpix object is deleted or its stoprender method is called.
.. ok, now I got it. I think that if you create the left render chain first it may work out, thats why the problem only showed up sometimes, really great that you were able to figure this out !
I am just looking at the chaining setup (I did this the last time some years ago, forgot completely how it worked).
Doing that I am trying to figure out how to reconstruct render chains every time the patch is changed, ... would be great if there was immediate feedback if the patch is edited.
Looking through the per-input code above, the obvious question is "how to make this easily extend to n-input objects?". The motivation for n-input objects seems pretty clear: e.g doing n layer compositing would be much neater and faster that way.
wait, ... I think that the fastest way of doing layers in Gem is still with alpha blending on several objects. Well, I see the usefulness of providing both possiblities though, and I think that in several situations the OpenGL way is not possible.
Another topic with pixes it feedback loops. I experimented with that some time ago, and it is possible with pix_buf, but this concept isn't really thought out yet.
I have already asked Johannes about setting up a CVS for GEM, this would make shared development a lot easier.
Yes, that would be good. I note that there's already a CVS archive for the GEM OSX project so it would be good to get a central repository up to keep things consistent and organised.
yep .. I am localy working with CVS too ...
Can you send a diff against gem-0.87 of the changes you did ?
thanks a lot .. just about to try it out ...
Guenter
From: g|nter geiger [mailto:geiger@xdv.org] Sent: Friday, 14 June 2002 7:30 PM
On Fri, 14 Jun 2002, Daniel Heckenberg wrote:
That's good news: those pix_es could definitely be optimized and MMX and Altivec are great for image processing. How far are you through this process?
Thats a good questions, I have only started. So far I have done the simple things like pix_add, pix_sub which got a tremendous speedup.
I have been working with pix_convolve a bit, no asm yet, but by specializing to fixed convolution kernels and avoiding allocation/deallocation of memory as well as floating point calculations I got a 2-3 fold improvement in speed. The asm code is a bit more complex there and I am not a specialist in doing this.
This does get hairy very quickly... in many situations with recent Intel processors it's faster to use floating point math than integer. MMX tips the balance back (particularly for image processing because there's native saturation mode arithmetic) but SSE gives you SIMD for floating point as well.
Intel have some highly optimized libraries for image processing (including 2d convolution) which are available for windows and linux. The licensing terms may be incompatible with pd (not sure about this) but they would provide a good performance reference.
Finally we have to check the pix_ objects step by step,and of course non asm version have to be supplied to .... maybe investigate on libraries for doing that.
There are problems in the release code when an dualpix object
is deleted or
its stoprender method is called.
.. ok, now I got it. I think that if you create the left render chain first it may work out, thats why the problem only showed up sometimes, really great that you were able to figure this out !
I am just looking at the chaining setup (I did this the last time some years ago, forgot completely how it worked).
Doing that I am trying to figure out how to reconstruct render chains every time the patch is changed, ... would be great if there was immediate feedback if the patch is edited.
Yes... I guess this would involve sending messages down the pd connections and comparing the resultant chain to the existing render chain. I'm not sure how slow this would be, altho it wouldn't be hard to check. Could be nice as an "edit" mode rather than "performance" mode.
Daniel
On Mon, 17 Jun 2002, Daniel Heckenberg wrote:
This does get hairy very quickly... in many situations with recent Intel processors it's faster to use floating point math than integer. MMX tips the balance back (particularly for image processing because there's native saturation mode arithmetic) but SSE gives you SIMD for floating point as well.
mhm, I thought integer would be faster in any case, and then there is the problem of memory usage, so there is no way of storing the data in float, because it would need 4 times more memory, hence conversion from float to integer, etc ... but I didnt look at what SSE offer for this.
Intel have some highly optimized libraries for image processing (including 2d convolution) which are available for windows and linux. The licensing terms may be incompatible with pd (not sure about this) but they would provide a good performance reference.
yes surely, actually I think they have the convolution as an example in there documentation.
As a reference, definitively very good, ... now someone has to hack it in :)
Doing that I am trying to figure out how to reconstruct render chains every time the patch is changed, ... would be great if there was immediate feedback if the patch is edited.
Yes... I guess this would involve sending messages down the pd connections and comparing the resultant chain to the existing render chain. I'm not sure how slow this would be, altho it wouldn't be hard to check. Could be nice as an "edit" mode rather than "performance" mode.
I think it will be fast enough ...
Guenter
On Fri, 14 Jun 2002, Daniel Heckenberg wrote:
To solve this I added a stoprender function to the render and postrender functions that are kept in the DAG list. Now a per-input stoprender function can simply clear that input's cache and if/when the object is deleted the unbroken DAGs can be properly broken. There may well be a better/more elegant way to do this, but this seems to fit with the present code.
The other obvious alternative is to pass the message down the PD connections (as happens when the DAG is constructed).
I am not yet really into the source, but a feeling tells me that the message passing solution might be more the way it should be. The "render chain" or dag (what does this stand for ?) commands should probably really only do the rendering... have to look at it further, maybe Mark or Johannes can tell you more about this.
There's another problem in the GemPixDualObj class, in its RightRender method. GemPixDualObj copies the GemState->image pointer to use later when its left render occurs.
This fails in the following case: If there are pix_fx objects in this right gemlist, they will reset the GemState->image pointer back to their "oldImage" values when their postrender functions are called. That means that when the GemPixDualObj Render occurs, its m_pixRight member will not be pointing to the correct image. This can cause strange problems like the [pix_data] problem that I reported, as well as more catastrophic problems if the image format is converted upstream (ie if there is a [pix_rgba] in the fx chain that is effectively rolled back before the GemPixDualObj occurs.)
As the pointer copy occurs regardless of the newImage flag, another problem arises: If pix_fx objects are in a chain and going into the right input of a GemPixDualObj and the render chain is running without the newImag flag set, the fx won't run but the image will be taken anyway.
Hmm, problem I may be solved by doing the postrender call after all gemheads have been rendered, and not on a per render chain. This would break rendering with a "bang" on the gemhead though.
Probably it would make sense to hook all objets into one gemhead, but then ordering would not work anymore, so we need a "gemtrigger"... ... finally I think I might talk complete nonsense ...
Looking through the per-input code above, the obvious question is "how to make this easily extend to n-input objects?". The motivation for n-input objects seems pretty clear: e.g doing n layer compositing would be much neater and faster that way.
ah, forget about the point I was making in my last mail about that, there are probably hundreds of thinkable effects that just might use three image or more as an input.
making image processing work in a similar way that signal processing is done in pd would be really cool. Having a basic set of objects to build homegrown video effects ...
Guenter
-----Original Message----- From: g|nter geiger [mailto:geiger@xdv.org] Sent: Saturday, 15 June 2002 12:49 AM
On Fri, 14 Jun 2002, Daniel Heckenberg wrote:
To solve this I added a stoprender function to the render and postrender functions that are kept in the DAG list. Now a per-input stoprender function can simply clear that input's cache and if/when the object is deleted the unbroken DAGs can be properly broken. There may well be a better/more elegant way to do this, but this seems to fit with
the present
code.
The other obvious alternative is to pass the message down the
PD connections
(as happens when the DAG is constructed).
I am not yet really into the source, but a feeling tells me that the message passing solution might be more the way it should be. The "render chain" or dag (what does this stand for ?) commands should probably really only do the rendering... have to look at it further, maybe Mark or Johannes can tell you more about this.
Yup. It would seem to be easier and more elegant to do it that way: e.g. use the same message that creates the chain but with a NULL cache pointer... but as GEM doesn't know about PD connection making or breaking, I think that the connections could change between when the render chain is made and when the chain is destroyed. I haven't looked into this in detail but it seems more robust and straightforward to use the existing chain itself to do the stoprender.
dag = directed acyclic graph (from memory).
Looking through the per-input code above, the obvious question
is "how to
make this easily extend to n-input objects?". The motivation for n-input objects seems pretty clear: e.g doing n layer compositing would be much neater and faster that way.
ah, forget about the point I was making in my last mail about that, there are probably hundreds of thinkable effects that just might use three image or more as an input.
making image processing work in a similar way that signal processing is done in pd would be really cool. Having a basic set of objects to build homegrown video effects ...
Yes - it would be great.
There are a couple of related projects (of which you're probably aware) for video in jMax: GridFlow and VideoDSP. If/when I get a chance I'll look at some of the image processing objects for jMax to see if there's any chance to make cross-development easy between those objects and the GEM pix_s. It would be good to share objects across the different systems and share the task of optimising things.
Daniel
On Mon, 17 Jun 2002, Daniel Heckenberg wrote:
dag = directed acyclic graph (from memory).
yes, makes sense to me.
There are a couple of related projects (of which you're probably aware) for video in jMax: GridFlow and VideoDSP. If/when I get a chance I'll look at some of the image processing objects for jMax to see if there's any chance to make cross-development easy between those objects and the GEM pix_s. It would be good to share objects across the different systems and share the task of optimising things.
Yes, I know the projects, but I haven't taken a look at them yet, should do so, though ...
Guenter
-
Daniel Heckenberg -
g�nter geiger