Hi,
I have to perform some very very simple computer-vision tasks (such as motion detection) on image data captured by a camera, and I was considering/evaluating the possibility of using GEM to do that.
It would involve basically computing the difference between two frames, background subtraction, masking an image with another one, computing the avarage value over a rectangular portion of the image, and this kind of things.
I'm a bit lost, since I guess there are probably facilities to do such things in GEM, but I don't know where to look for.
- Are there some pix_ objects to do these kinds of operations in a
reasonably efficient way? 2) Is it possible at all to use the openGL wrappers to do such calculations in the GPU rather than CPU? Where may I find examples of that? 3) Are there some objects that already implement the latter approach and offer basic functions at a slightly higher level, such as frame-difference on a texture buffer?
If anyone can point me to any documentation/examples/tutorials specific to this kind of things, I'll be very grateful.
Or maybe this is not the kind of things GEM is suitable for? Should I forget about it and have a look at other platforms?
Thanks, m.
See the attached patch, Sara Kolster and I use it in our workshops. Original by Ben Bogart, posted to the list when we had similar questions a few years ago.
best, d.
matteo sisti sette wrote:
I have to perform some very very simple computer-vision tasks (such as motion detection) on image data captured by a camera, and I was considering/evaluating the possibility of using GEM to do that.
Thanks a lot, very interesting!!
Just a quetion/problem. I'm trying your attached patch, and it works, but the Y position of the cursor seems to be systematically inverted!
I mean, if the actual blob (as I can see it in the preview "window") is on the top, the circle is drawn on the bottom of the screen!!
Is it an error in the patch, or is something working wrong in my case?
Aslo, the comments about "pix_blob" output say: "[upper left corner is 1, 1 down right corner is 0, 0]" According to what I can see by attaching number boxes to the 2nd and 3rd outlet, I would say "[upper left corner is 0, 0 down right corner is 1, 1]"
2008/1/21, Derek Holzer derek@umatic.nl:
See the attached patch, Sara Kolster and I use it in our workshops. Original by Ben Bogart, posted to the list when we had similar questions a few years ago.
best, d.
matteo sisti sette wrote:
I have to perform some very very simple computer-vision tasks (such as motion detection) on image data captured by a camera, and I was considering/evaluating the possibility of using GEM to do that.
-- derek holzer ::: http://www.umatic.nl ::: http://blog.myspace.com/macumbista ---Oblique Strategy # 58: "Do we need holes?"
#N canvas 574 31 1163 721 10; #X obj 277 131 pix_grey; #X msg 302 159 reset; #X obj 277 188 pix_background; #X msg 370 167 $1 $1 $1; #X msg 370 132 0.5; #X floatatom 370 151 5 0 0 0 - - -; #N canvas 486 104 585 607 cursor 0; #X obj 19 36 inlet; #X obj 231 32 inlet; #X obj 151 346 translateXYZ; #X text 62 150 0.64; #X text 34 151 1; #X obj 20 63 t f f; #X obj 34 176 -; #X text 59 177 X-Range; #X text 280 176 Y-Range; #X obj 53 215 expr (8/$f1); #X obj 20 215 -; #X obj 38 241 *; #X obj 26 95 max 0; #X obj 68 95 min 10; #X obj 58 131 f; #X obj 30 131 f; #X obj 39 267 - 4; #X text 278 148 0.64; #X text 250 147 1; #X obj 232 60 t f f; #X obj 250 172 -; #X obj 267 215 expr (8/$f1); #X obj 233 215 -; #X obj 251 241 *; #X obj 238 92 max 0; #X obj 280 92 min 10; #X obj 272 128 f; #X obj 246 127 f; #X obj 221 268 expr (-1*$f1)+4; #X obj 436 30 inlet; #X obj 437 61 * 10; #X obj 150 323 gemhead 15; #X obj 357 496 circle 0.1; #X obj 33 410 separator; #X obj 204 410 separator; #X obj 205 496 rectangle 0.2 0.01; #X obj 357 410 separator; #X msg 367 454 draw line; #X obj 367 433 loadbang; #X obj 152 371 colorRGB 1 0 0 1; #X obj 32 495 rectangle 0.01 0.2; #X msg 438 454 width $1; #X obj 437 398 int; #X obj 438 425 + 1; #X text 433 10 size; #X text 20 13 X-coordinate; #X text 221 11 Y-coordinate; #X text 278 370 color red; #X connect 0 0 5 0; #X connect 1 0 19 0; #X connect 2 0 39 0; #X connect 5 0 10 0; #X connect 5 0 12 0; #X connect 5 1 13 0; #X connect 6 0 9 0; #X connect 9 0 11 1; #X connect 10 0 11 0; #X connect 11 0 16 0; #X connect 12 0 15 0; #X connect 13 0 14 0; #X connect 14 0 6 1; #X connect 14 0 10 1; #X connect 14 0 13 1; #X connect 15 0 6 0; #X connect 15 0 12 1; #X connect 16 0 2 1; #X connect 19 0 22 0; #X connect 19 0 24 0; #X connect 19 1 25 0; #X connect 20 0 21 0; #X connect 21 0 23 1; #X connect 22 0 23 0; #X connect 23 0 28 0; #X connect 24 0 27 0; #X connect 25 0 26 0; #X connect 26 0 20 1; #X connect 26 0 22 1; #X connect 26 0 25 1; #X connect 27 0 20 0; #X connect 27 0 24 1; #X connect 28 0 2 2; #X connect 29 0 30 0; #X connect 30 0 32 1; #X connect 30 0 42 0; #X connect 31 0 2 0; #X connect 33 0 40 0; #X connect 34 0 35 0; #X connect 36 0 32 0; #X connect 37 0 32 0; #X connect 38 0 37 0; #X connect 39 0 33 0; #X connect 39 0 34 0; #X connect 39 0 36 0; #X connect 41 0 32 0; #X connect 42 0 43 0; #X connect 43 0 41 0; #X restore 282 510 pd cursor; #X obj 612 110 separator; #X obj 612 131 pix_texture; #X obj 612 152 square 4; #X obj 62 264 pix_texture; #X obj 62 244 alpha; #X obj 62 224 colourRGB 1 1 1 0.65; #X obj 62 284 translateXYZ -2.4 -2.4 0; #X obj 427 49 gemhead 10; #X obj 62 205 spigot; #X obj 99 160 tgl 20 0 empty empty empty 0 -6 0 8 -262144 -1 -1 1 1 ; #X obj 427 70 pix_video; #X obj 16 114 gemwin 30; #X msg 24 47 create; #X msg 25 71 destroy; #X obj 19 18 tgl 15 0 empty empty empty 0 -6 0 8 -262144 -1 -1 1 1 ; #X text 335 440 size; #X text 315 440 Y; #X text 289 439 X; #X text 859 10 by Ben Bogart (ben@ekran.org); #X obj 62 304 square 1.5; #X obj 277 106 separator; #X obj 278 423 pix_blob; #X obj 205 15 loadbang; #X msg 202 34 0; #X obj 426 15 tgl 15 0 empty empty empty 0 -6 0 8 -262144 -1 -1 1 1 ; #X msg 33 92 dimen 300 300; #X text 48 16 gem window; #X text 449 13 live input camera [firewire/usb]; #X text 610 224 [pix_background] 'removes' the background of an image by comparing a static image in memory to an incoming video stream. all values withing a given range are turned black. for best effect , place a camera on a static scene like a blank wall and hit 'reset'. then set the range control to cover any variance in the background. when a new object enters the scene it will be the only thing draw against a black background. use pix_chroma_key or pix_compare to add in images in place of the black.; #X text 611 199 --------------; #X text 607 85 image of the camera; #X text 608 93 --------------------; #X text 610 392 Inlet 2: float: range (<f> == <f> <f> <f>); #X text 609 341 Inlet 2: list: range ; in RGBA mode this is <+-red> <+-green> <+-blue> ; in YUV-mode this is <+-luma> <+-Cb> <+-Cr> ; in Gray-mode only the first value is important <+-gray>; #X text 612 211 Description: separate an object from background; #X text 215 80 motion tracking; #X text 216 88 ---------------; #X text 34 167 --------; #X text 440 171 threshold; #X text 605 446 [pix_blob] will calculate the "center-of-gravity" of a certain (combination of) channel(s).; #X text 604 434 --------------; #X text 607 476 You can choose the combination of channels used for the calculation by changing the <mode>. Valid modes are: 0(gray)(default) , 1(red) , 2(green) , 3(blue) , 4(alpha); #X text 608 520 pE: selecting "red" (1) will weight position of each pixel with its red-value. Pixels with "more" red will be weighted stronger , thus moving the "center of gravity" -- the blob position -- nearer to themself. The total amount of "red" in the image defines the size of the blob.; #X text 608 587 You can specify a custom combination of channels with <color-weights>. pE: "1 0 2" will give more weight to the blue-channel than to the red-channel , green- and alpha-values will be ignored. ; #X text 608 642 Specifing only one single color-weight <w> , is the same as "<w> <w> <w> 0"; #X text 387 190 ------------------------------- PIX_BACKGROUND; #X text 380 424 ------------------------------- PIX_BLOB; #X text 858 23 changed by Sara Kolster; #X text 857 36 [text from GEM help patches]; #X msg 304 373 <mode>; #X msg 329 395 <color-weights>; #X text 273 529 representation of the 'movement'; #X text 274 541 by a cursor. Click on pd cursor; #X text 273 553 to open the subpatch; #X text 37 441 outlet 1: gemlist; #X text 37 482 outlet 4: size; #X text 37 468 outlet 3: Y coordinate; #X text 37 454 outlet 2: X coordinate; #X text 35 496 [upper left corner is 1 , 1; #X text 36 509 down right corner is 0 , 0]; #X text 35 430 ----------------; #X text 37 421 OUTPUT PIX_BLOB ---------------; #X text 34 159 preview; #X obj 319 668 send tracking; #X obj 372 594 change; #X text 317 686 send coordinates to soundpatch; #X text 856 66 1] make gemwindow; #X text 856 79 2] turn on rendering; #X text 856 92 3] set the background; #X text 857 116 5] move object and receive information; #X text 856 104 4] turn on live input camera; #X msg 372 615 s $1; #X msg 331 615 y $1; #X msg 292 615 x $1; #X msg 110 571 send x $1; #X msg 153 595 send y $1; #X msg 178 618 send s $1; #X msg 51 685 connect localhost 3000; #X obj 124 643 netsend; #X connect 0 0 2 0; #X connect 1 0 2 0; #X connect 2 0 15 0; #X connect 2 0 28 0; #X connect 3 0 2 1; #X connect 4 0 5 0; #X connect 5 0 3 0; #X connect 7 0 8 0; #X connect 8 0 9 0; #X connect 10 0 13 0; #X connect 11 0 10 0; #X connect 12 0 11 0; #X connect 13 0 26 0; #X connect 14 0 17 0; #X connect 15 0 12 0; #X connect 16 0 15 1; #X connect 17 0 7 0; #X connect 17 0 27 0; #X connect 19 0 18 0; #X connect 20 0 18 0; #X connect 21 0 18 0; #X connect 27 0 0 0; #X connect 28 1 6 0; #X connect 28 1 80 0; #X connect 28 1 81 0; #X connect 28 2 6 1; #X connect 28 2 79 0; #X connect 28 2 82 0; #X connect 28 3 6 2; #X connect 28 3 71 0; #X connect 28 3 83 0; #X connect 29 0 30 0; #X connect 30 0 14 0; #X connect 31 0 14 0; #X connect 32 0 18 0; #X connect 56 0 28 1; #X connect 57 0 28 2; #X connect 71 0 78 0; #X connect 78 0 70 0; #X connect 79 0 70 0; #X connect 80 0 70 0; #X connect 81 0 85 0; #X connect 82 0 85 0; #X connect 83 0 85 0; #X connect 84 0 85 0;
Hi Matteo,
I don't have a camera to plug in and double check, but I never noticed this inverted problem before. You can always re-invert the numbers if you need to ;-)
Anyways, inverted or not, the patch does what it's supposed to, I hope!
best, d.
matteo sisti sette wrote:
Thanks a lot, very interesting!!
Just a quetion/problem. I'm trying your attached patch, and it works, but the Y position of the cursor seems to be systematically inverted!
I mean, if the actual blob (as I can see it in the preview "window") is on the top, the circle is drawn on the bottom of the screen!!
Is it an error in the patch, or is something working wrong in my case?
Aslo, the comments about "pix_blob" output say: "[upper left corner is 1, 1 down right corner is 0, 0]" According to what I can see by attaching number boxes to the 2nd and 3rd outlet, I would say "[upper left corner is 0, 0 down right corner is 1, 1]"
I don't have a camera to plug in and double check, but I never noticed this inverted problem before. You can always re-invert the numbers if you need to ;-)
Yes of course it is stright-forward to correct the inversion. But I was wondering if there may be some weird platform-dependence in the involved objects that may cause the y coordinate to work in one sense or the other in different computers/os's?? Because it would be really really strange that it were just an error in the patch and you never noticed before - I mean the cursor moves down when the object moves up, and you mention having used it in workshops for years.
I was just curious if it may be some platform or version dependency.
I use Windows XP. Here, the Y output of pix_block is 0 when the blob is on the bottom, and 1 when it is on the top.
But the patch is great, helped me a lot and is just (part of) what I was looking for.
(btw, in my previous message I stated it wrong: I get 0,0 at bottom-left and 1,1 at top-right, which is neither what I said nor what is written in the patch comments)(I call left and right with respect to the image being processed, not the user looking at himself on the screen)
Anyways, inverted or not, the patch does what it's supposed to, I hope!
best, d.
matteo sisti sette wrote:
Thanks a lot, very interesting!!
Just a quetion/problem. I'm trying your attached patch, and it works, but the Y position of the cursor seems to be systematically inverted!
I mean, if the actual blob (as I can see it in the preview "window") is on the top, the circle is drawn on the bottom of the screen!!
Is it an error in the patch, or is something working wrong in my case?
Aslo, the comments about "pix_blob" output say: "[upper left corner is 1, 1 down right corner is 0, 0]" According to what I can see by attaching number boxes to the 2nd and 3rd outlet, I would say "[upper left corner is 0, 0 down right corner is 1, 1]"
-- derek holzer ::: http://www.umatic.nl ::: http://blog.myspace.com/macumbista ---Oblique Strategy # 76: "Give the game away"
matteo sisti sette wrote:
I don't have a camera to plug in and double check, but I never noticed this inverted problem before. You can always re-invert the numbers if you need to ;-)
Yes of course it is stright-forward to correct the inversion. But I was wondering if there may be some weird platform-dependence in the involved objects that may cause the y coordinate to work in one sense or the other in different computers/os's?? Because it would be really really strange that it were just an error in the patch and you never noticed before - I mean the cursor moves down when the object moves up, and you mention having used it in workshops for years.
I was just curious if it may be some platform or version dependency.
the problem is, that some pix-sources (platform dependent!) supply the images with (0/0) in the upper-left corner, whereas others (0/0) is the lower-left corner (like openGL expects it). for performance reasons, Gem does not put the images "right" when it retrieves them, but tries to flip the image on the texture, once they are uploaded. this often makes things behave weird (e.g. mixing 2 images will make one of them upside-down, even though the look "normal" when viewed separately) so, sometimes the upside-downness is not respected by certain pix_objects (it seems like [pix_blob] is one of them), this is a bug and should be fixed.
finally, i am wondering how much the penalty would be to manually flip the images when retrieving them (this would involve one additional copy of the entire image, each time it is pulled from the source) - this will just magically fix a lot of weirdness...
masd.r IOhannes
Hi IOhannes,
IOhannes m zmoelnig wrote:
the problem is, that some pix-sources (platform dependent!) supply the images with (0/0) in the upper-left corner, whereas others (0/0) is the lower-left corner (like openGL expects it).
Would be good to know which platforms do it right, and which ones wrong. Let me guess.....
d.
On Jan 21, 2008 9:20 AM, Derek Holzer derek@umatic.nl wrote:
Hi IOhannes,
IOhannes m zmoelnig wrote:
the problem is, that some pix-sources (platform dependent!) supply the images with (0/0) in the upper-left corner, whereas others (0/0) is the lower-left corner (like openGL expects it).
Would be good to know which platforms do it right, and which ones wrong. Let me guess.....
It is actually the APIs which handle the images and video. Quicktime has the origin at the upper left while DirectShow is the lower left.
Derek Holzer wrote:
Hi IOhannes,
IOhannes m zmoelnig wrote:
the problem is, that some pix-sources (platform dependent!) supply the images with (0/0) in the upper-left corner, whereas others (0/0) is the lower-left corner (like openGL expects it).
Would be good to know which platforms do it right, and which ones wrong.
it's not a matter of "right" and "wrong".
mathematicians tend to put the origin in the lower-left corner (like a coordinate system) computer-graphics prefer the upper-left corner (back in the old days, the cursor would appear in the 1st line on the screen)
openGL has been designed by mathematicians. tcl/tk comes from the computer geeks...
btw, this fuzz is the reason why the tables in Pd are never oriented the right way...
mgfa.dr IOhannes
matteo:
try this:
http://hangar.org/wikis/lab/doku.php?id=start:puredata_opencv
palmieri
2008/1/21 IOhannes m zmoelnig zmoelnig@iem.at:
Derek Holzer wrote:
Hi IOhannes,
IOhannes m zmoelnig wrote:
the problem is, that some pix-sources (platform dependent!) supply the images with (0/0) in the upper-left corner, whereas others (0/0) is the lower-left corner (like openGL expects it).
Would be good to know which platforms do it right, and which ones wrong.
it's not a matter of "right" and "wrong".
mathematicians tend to put the origin in the lower-left corner (like a coordinate system) computer-graphics prefer the upper-left corner (back in the old days, the cursor would appear in the 1st line on the screen)
openGL has been designed by mathematicians. tcl/tk comes from the computer geeks...
btw, this fuzz is the reason why the tables in Pd are never oriented the right way...
mgfa.dr IOhannes
PD-list@iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
On Jan 21, 2008 9:07 AM, IOhannes m zmoelnig zmoelnig@iem.at wrote:
finally, i am wondering how much the penalty would be to manually flip the images when retrieving them (this would involve one additional copy of the entire image, each time it is pulled from the source) - this will just magically fix a lot of weirdness...
For high definition video the performance hit would potentially be great. Since the memory will be accessed backwards the cache streaming will be poor too.
chris clepper wrote:
On Jan 21, 2008 9:07 AM, IOhannes m zmoelnig <zmoelnig@iem.at mailto:zmoelnig@iem.at> wrote:
finally, i am wondering how much the penalty would be to manually flip the images when retrieving them (this would involve one additional copy of the entire image, each time it is pulled from the source) - this will just magically fix a lot of weirdness...For high definition video the performance hit would potentially be great.
yes, i am aware of that, so i don't think that we should just go and replace the things as they currently are.
i am more and more fond of having a gem-settings mechanism (like .pdrc), where you (the user) could set such things to you personal taste.
Since the memory will be accessed backwards the cache streaming will be poor too.
hmm, would there be a difference once the images are converted? that is: do image-sources with origin in lower-left perform bad on os-x?
fgmasdr. IOhannes
-
chris clepper -
Derek Holzer -
IOhannes m zmoelnig -
matteo sisti sette -
palmieri, ricardo