Hello,
Let me start off by saying that I don't have much experience with computer graphics, so my knowledge is very limited on this area, I'm more of an audio guy but I have some ideas in mind that would apply to graphics well. I just want to experiment.
I want to give a simple example to show my question. Suppose I have a video running and I want to delay each pixel between 0 and a maximum time seperately. This is the most basic time based effect I can think of. So for a 320x200 video for example, I would need 64000 delaying units running seperately and to be able to do this in realtime, I think I'd need to use GPU for computation.
So I'm thinking of using pixel shaders. But I'm not really sure if this is possible or not with it. I grabbed the orange book to have an idea about the process, I did not have much time tolook in detail but I could not find any references to such an operation that made me think that I'm on the wrong track. So before going any further learning GLSL, I'd like to have your ideas on this.
Is GLSL along with GEM is a nice way to do such an operation? Is it even possible to do it in realtime? If GPU powered realtime operation is not possible, is there any tool that you know that is capable of doing such things(realtime or offline)? I have many ideas on processes that modifies pixels which are dependant on the states of pixels before them(i.e. with memory) and I'm trying to find a way to implement them. Any help is appreciated.
Thanks BB
This is entirely possible, however you would want to use a 3D texture,
something on the order of dimensions 320 by 240 by x (where x is how
many frames of time 'back' you want to go). This will be relatively
heavy video memory wise I would imagine. However, I have a shader for
you that does this.
Absolutely no idea what so over if GEM can deal with 3D textures. Id
assume since it has low level openGl support it should be able to, but
how to get 2D video frames into that 3D texture via pix_xxx I have no
idea.
Here is the shader, written by Andrew Benson from Cycling74 iirc.
This is sans vertex shader, but the vertex shader is basically a
passthrough, so its very simple.
texa is a 2d lookup table - greyscale, where the luminance of the
pixel at the point determines how far back in time to go. This shader
assumes a 512x512 map, but you could make that dynamic by passing a
varying variable from the vertex shader that looks at the dims of the
3D texture.
HTH.
varying vec2 texcoord0; varying vec2 texcoord1; uniform float slice; uniform sampler3D texo; uniform sampler2DRect texa; const vec4 coeff = vec4(0.299, .587, 0.114, 0.);
void main( void ) { float v1 = dot(texture2DRect(texa,texcoord0*vec2(512.)),coeff); //assumes 512 x 512 slice map. Pretty arbitrary... vec4 v0 = texture3D(texo, vec3(texcoord0.xy,v1)); gl_FragColor = v0; }
On Jan 6, 2008, at 4:21 AM, Batuhan Bozkurt wrote:
Hello,
Let me start off by saying that I don't have much experience with computer graphics, so my knowledge is very limited on this area, I'm more of an audio guy but I have some ideas in mind that would apply to graphics well. I just want to experiment.
I want to give a simple example to show my question. Suppose I have a video running and I want to delay each pixel between 0 and a maximum time seperately. This is the most basic time based effect I can think of. So for a 320x200 video for example, I would need 64000 delaying units running seperately and to be able to do this in realtime, I
think I'd need to use GPU for computation.So I'm thinking of using pixel shaders. But I'm not really sure if
this is possible or not with it. I grabbed the orange book to have an idea about the process, I did not have much time tolook in detail but I
could not find any references to such an operation that made me think that
I'm on the wrong track. So before going any further learning GLSL, I'd
like to have your ideas on this.Is GLSL along with GEM is a nice way to do such an operation? Is it
even possible to do it in realtime? If GPU powered realtime operation is not possible, is there any tool that you know that is capable of doing such things(realtime or
offline)? I have many ideas on processes that modifies pixels which are
dependant on the states of pixels before them(i.e. with memory) and I'm trying
to find a way to implement them. Any help is appreciated.Thanks BB
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
This is a very elegant solution for a delay effect I should say. I've never known something called "3d texture" exists but it completely makes sense. I've searched through documentation and the list for gem support for such thing but no luck yet. But thanks for the shader and the idea anyways!
The problem here is that, I actually need a more generalised framework of working with time and past frames(or pixels) when dealing with time based effects. This vertex shader approach with 3D texture and a 2D map probably would not help let's say, if I wanted to use an averaging lowpass filter(depending on past pixel values) to the R G and B values of each pixel seperately.
I'd like to know if there is any way to hold values back in buffers(like arrays) on shader level and make operations on that data by shader programming. So maybe I could say "get current value and average it with the previous value and set it as the new current value" or something like that. If there is a way, than I'd like to dig more deeply into it, but if this question looks very stupid than I'm probably getting the concept of graphics pipeline and purpose of shading language completely wrong so maybe I should stop wasting time on thinking like this and try to find other approaches.
BB
vade wrote:
This is entirely possible, however you would want to use a 3D texture, something on the order of dimensions 320 by 240 by x (where x is how many frames of time 'back' you want to go). This will be relatively heavy video memory wise I would imagine. However, I have a shader for you that does this.
Absolutely no idea what so over if GEM can deal with 3D textures. Id assume since it has low level openGl support it should be able to, but how to get 2D video frames into that 3D texture via pix_xxx I have no idea.
Here is the shader, written by Andrew Benson from Cycling74 iirc.
This is sans vertex shader, but the vertex shader is basically a passthrough, so its very simple.
texa is a 2d lookup table - greyscale, where the luminance of the pixel at the point determines how far back in time to go. This shader assumes a 512x512 map, but you could make that dynamic by passing a varying variable from the vertex shader that looks at the dims of the 3D texture.
HTH.
varying vec2 texcoord0; varying vec2 texcoord1; uniform float slice; uniform sampler3D texo; uniform sampler2DRect texa; const vec4 coeff = vec4(0.299, .587, 0.114, 0.);
void main( void ) { float v1 = dot(texture2DRect(texa,texcoord0*vec2(512.)),coeff); //assumes 512 x 512 slice map. Pretty arbitrary... vec4 v0 = texture3D(texo, vec3(texcoord0.xy,v1)); gl_FragColor = v0; }
On Jan 6, 2008, at 4:21 AM, Batuhan Bozkurt wrote:
Hello,
Let me start off by saying that I don't have much experience with computer graphics, so my knowledge is very limited on this area, I'm more of an audio guy but I have some ideas in mind that would apply to graphics well. I just want to experiment.
I want to give a simple example to show my question. Suppose I have a video running and I want to delay each pixel between 0 and a maximum time seperately. This is the most basic time based effect I can think of. So for a 320x200 video for example, I would need 64000 delaying units running seperately and to be able to do this in realtime, I think I'd need to use GPU for computation.
So I'm thinking of using pixel shaders. But I'm not really sure if this is possible or not with it. I grabbed the orange book to have an idea about the process, I did not have much time tolook in detail but I could not find any references to such an operation that made me think that I'm on the wrong track. So before going any further learning GLSL, I'd like to have your ideas on this.
Is GLSL along with GEM is a nice way to do such an operation? Is it even possible to do it in realtime? If GPU powered realtime operation is not possible, is there any tool that you know that is capable of doing such things(realtime or offline)? I have many ideas on processes that modifies pixels which are dependant on the states of pixels before them(i.e. with memory) and I'm trying to find a way to implement them. Any help is appreciated.
Thanks BB
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
hi batuhan
i was once working in this area: http://romanhaefeli.net/projekte/out_of_frame/
however, at this time i had troubles implementing the idea in a way, so that movies with decent resolution and quality could be rendered. my first try of an implemention did indeed use gem, but in a probably not so elegant way: i used [pix_pix2sig~] to convert the incoming live video stream into an audio-signal, so that i could use tables to store the pixel data. this allowed to random-access particular pixels of particular frames very easily. the patch was very slow and in realtime i could only run 160x120 px at 30fps.
then i decided to go for a python script in order to apply the process offline on prerecorded footage. but also the python script was _very_ slow (rendering 30s of dv-pal video took ~30min), but i didn't care since i could do it now in arbitrary quality/resolution. the videos from the link above are all done using this python script.
after that i started to learn gridflow and encountered, that the implementation of this method is much more straighforward with gridflow than with gem or python (yeah, i know it is kind of stupid to use an interpreted language like python for such an iterative process [720 * 576 * 3subpx * 25fps * 30 s = 933120000 iterations of the very same algorithm for a 30 second movie]. i wouldn't be surprised, if implemented in c it would run in realtime, but i completely lack any c skills). in gridflow you only need a handful of objects and it is faster than my gem-pix-to-sig-to-pix approach and python, though it is still quite slow. i can now run 320x240 * 20fps in realtime. but more important: with gridflow it's quite simple to do and very flexible. ask me again, if you want to see the gridflow patch. a few examples using gridflow in realtime can be shown here: http://romanhaefeli.net/kanada/filmchen/
if you manage to find an implementation that is baed on the gpu, please let us know. at least i would be very much interested.
good luck ;-) roman
On Sun, 2008-01-06 at 11:21 +0200, Batuhan Bozkurt wrote:
Hello,
Let me start off by saying that I don't have much experience with computer graphics, so my knowledge is very limited on this area, I'm more of an audio guy but I have some ideas in mind that would apply to graphics well. I just want to experiment.
I want to give a simple example to show my question. Suppose I have a video running and I want to delay each pixel between 0 and a maximum time seperately. This is the most basic time based effect I can think of. So for a 320x200 video for example, I would need 64000 delaying units running seperately and to be able to do this in realtime, I think I'd need to use GPU for computation.
So I'm thinking of using pixel shaders. But I'm not really sure if this is possible or not with it. I grabbed the orange book to have an idea about the process, I did not have much time tolook in detail but I could not find any references to such an operation that made me think that I'm on the wrong track. So before going any further learning GLSL, I'd like to have your ideas on this.
Is GLSL along with GEM is a nice way to do such an operation? Is it even possible to do it in realtime? If GPU powered realtime operation is not possible, is there any tool that you know that is capable of doing such things(realtime or offline)? I have many ideas on processes that modifies pixels which are dependant on the states of pixels before them(i.e. with memory) and I'm trying to find a way to implement them. Any help is appreciated.
Thanks BB
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
___________________________________________________________ Der frühe Vogel fängt den Wurm. Hier gelangen Sie zum neuen Yahoo! Mail: http://mail.yahoo.de
On Mon, 7 Jan 2008, Roman Haefeli wrote:
in gridflow you only need a handful of objects and it is faster than my gem-pix-to-sig-to-pix approach and python, though it is still quite slow. i can now run 320x240 * 20fps in realtime. but more important: with gridflow it's quite simple to do and very flexible.
I just thought of another way of doing slitscanning that runs much faster. You need to use a [#store] with "put_at" and write at a different row each time while reading back the image in whole rows. e.g. you send a (240,2)-sized grid to the left of the store. This avoids any clumsy copyings of the buffers. To make it work with columns, do it with rows but flip the image along the main diagonal using [#transpose].
I think that I'll try to do it in cellular_1d.pd soon, to replace the current junk.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal QC Canada
-
Batuhan Bozkurt -
Mathieu Bouchard -
Roman Haefeli -
vade