Dear IOhannes and gem community,
I have writen a color detector and would like to see if you think it could be included in Gem. As far as I researched in the past weeks, there is no similar code in Pd libraries. I developed this code by careful tuning based on a ground-truth dataset of about 200 color images from paper. The code detects color classes that is *perceptually* and unambiguously red, green, blue, yellow, black, and white. It will let you know when it is unsure of the output.
Perhaps you can pull the "color" branch from my published Git repo at:
git://github.com/rfabbri/Gem.git
I rebased the branch on the latest Gem git master branch.
I wrote a help patch help/pix_colordetect-help.pd which works on an image and an example/demo patch in examples/04.video/07.bg_subtract_tracking.pd that works on a video. To test the code, I suggest taking a look at these patches.
I look forward to your feedback, and your opinion about incorporating it into Gem. Ricardo Fabbri -- Linux registered user #175401 www.lems.brown.edu/~rfabbri labmacambira.sf.net
This sounds great! I know many people would love to use this, its a common question I get from students.
.hc
On Nov 9, 2011, at 5:47 PM, Ricardo Fabbri wrote:
Dear IOhannes and gem community,
I have writen a color detector and would like to see if you think it could be included in Gem. As far as I researched in the past weeks, there is no similar code in Pd libraries. I developed this code by careful tuning based on a ground-truth dataset of about 200 color images from paper. The code detects color classes that is *perceptually* and unambiguously red, green, blue, yellow, black, and white. It will let you know when it is unsure of the output.
Perhaps you can pull the "color" branch from my published Git repo at:
git://github.com/rfabbri/Gem.git
I rebased the branch on the latest Gem git master branch.
I wrote a help patch help/pix_colordetect-help.pd which works on an image and an example/demo patch in examples/04.video/07.bg_subtract_tracking.pd that works on a video. To test the code, I suggest taking a look at these patches.
I look forward to your feedback, and your opinion about incorporating it into Gem. Ricardo Fabbri -- Linux registered user #175401 www.lems.brown.edu/~rfabbri labmacambira.sf.net
GEM-dev mailing list GEM-dev@iem.at http://lists.puredata.info/listinfo/gem-dev
----------------------------------------------------------------------------
If you are not part of the solution, you are part of the problem.
On 11/09/2011 11:47 PM, Ricardo Fabbri wrote:
Perhaps you can pull the "color" branch from my published Git repo at:
git://github.com/rfabbri/Gem.git
looks good, but:
before including it, do you think there might be a better name than [pix_colordetect]? whenever i hear "detect" is assume a domain transformation from data into symbolic representation. e.g. [pix_movement] is not called [pix_motiondetect] because it stays in the data domain, just like your object. unfortunately [pix_colorreduce] is already taken (but then your object is more strict in what it can actually do)
ah, and would it make sense to encode the quality of the guess in the alpha-channel? (or did you plan to output a quality mask as a 2nd image?)
fgmasr IOhannes
Hello, IOhannes,
glad to hear it can be useful. It could also be called pix_colorclassify or pix_colorclass, since it outputs 6 color classes 'red', green', ..., 'white' (and a seventh 'not sure' class label). However, to me the output is in symbolic form. The 'red' symbol is encoded as [1 0 0] and so on. These data representtions re actually hard symbols for each color class. I wrote another object which ca get the output encoded in this way and generate 6 iages, one for each color class, e.g. a binary image indicating the presence of 'red' label pixels (which is not channel R since yellow also has R channel), another for 'yellow', etc.
Yes, the confidence could be in the alpha channel, although that could confuse visualizatin prior to decoding. The current output has no guesses; the beige color shows any pixels the lgorithm is unsure of. NOte that when it s eemingly makes mistakes, you can see in the original video that the color is actually right, its just that illumination direction and color balance actually distorts the color, and interreflections also chnage pixelvalues to something other than expected. The pS3eye, for instance, seems to have aslightly bluish white balance offset. I saw that usully the detection makes sense. However, red could be tuned to be slightly stricter, and similarly for some other colors.
In the future that would be a flag to also output color guesses where unsure. I could include that now. Moreover, yes, I want to output annimage or alpha mask encoding 3 levels of certainty: 0 uncertain, 1 good guess, and 2 certain.
What i was thinking bout the multiple image outlets was to decode the current 7-class output into 7 binary image outputs. however I wrote another object to do the decoding very naturally, avoiding wasting memory when e.g. just a couple of color classes are needed. that object called pix_equal, outputs a binary image with 255 where a pixel is within a given closed rgba interval on all rgba channels. Then the binary image can be fed to pix_multiblob for obtaining color blob parameters to control sound, etc. All that is already done with a demo. I just need to make pix_equal work for other image types than rgba prior to submitting it for your review.
best,
On Friday, November 11, 2011, IOhannes zmölnig zmoelnig@iem.at wrote:
On 11/09/2011 11:47 PM, Ricardo Fabbri wrote:
Perhaps you can pull the "color" branch from my published Git repo at:
git://github.com/rfabbri/Gem.git
looks good, but:
before including it, do you think there might be a better name than [pix_colordetect]? whenever i hear "detect" is assume a domain transformation from data into symbolic representation. e.g. [pix_movement] is not called [pix_motiondetect] because it stays in the data domain, just like your object. unfortunately [pix_colorreduce] is already taken (but then your object is more strict in what it can actually do)
ah, and would it make sense to encode the quality of the guess in the alpha-channel? (or did you plan to output a quality mask as a 2nd image?)
fgmasr IOhannes
sorry for the typos... writing from mobile..
On Friday, November 11, 2011, Ricardo Fabbri rfabbri@gmail.com wrote:
Hello, IOhannes,
glad to hear it can be useful. It could also be called pix_colorclassify
or pix_colorclass, since it outputs 6 color classes 'red', green', ..., 'white' (and a seventh 'not sure' class label). However, to me the output is in symbolic form. The 'red' symbol is encoded as [1 0 0] and so on. These data representtions re actually hard symbols for each color class. I wrote another object which ca get the output encoded in this way and generate 6 iages, one for each color class, e.g. a binary image indicating the presence of 'red' label pixels (which is not channel R since yellow also has R channel), another for 'yellow', etc.
Yes, the confidence could be in the alpha channel, although that could
confuse visualizatin prior to decoding. The current output has no guesses; the beige color shows any pixels the lgorithm is unsure of. NOte that when it s eemingly makes mistakes, you can see in the original video that the color is actually right, its just that illumination direction and color balance actually distorts the color, and interreflections also chnage pixelvalues to something other than expected. The pS3eye, for instance, seems to have aslightly bluish white balance offset. I saw that usully the detection makes sense. However, red could be tuned to be slightly stricter, and similarly for some other colors.
In the future that would be a flag to also output color guesses where
unsure. I could include that now. Moreover, yes, I want to output annimage or alpha mask encoding 3 levels of certainty: 0 uncertain, 1 good guess, and 2 certain.
What i was thinking bout the multiple image outlets was to decode the
current 7-class output into 7 binary image outputs. however I wrote another object to do the decoding very naturally, avoiding wasting memory when e.g. just a couple of color classes are needed. that object called pix_equal, outputs a binary image with 255 where a pixel is within a given closed rgba interval on all rgba channels. Then the binary image can be fed to pix_multiblob for obtaining color blob parameters to control sound, etc. All that is already done with a demo. I just need to make pix_equal work for other image types than rgba prior to submitting it for your review.
best,
On Friday, November 11, 2011, IOhannes zmölnig zmoelnig@iem.at wrote:
On 11/09/2011 11:47 PM, Ricardo Fabbri wrote:
Perhaps you can pull the "color" branch from my published Git repo at:
git://github.com/rfabbri/Gem.git
looks good, but:
before including it, do you think there might be a better name than [pix_colordetect]? whenever i hear "detect" is assume a domain transformation from data into symbolic representation. e.g. [pix_movement] is not called [pix_motiondetect] because it stays in the data domain, just like your object. unfortunately [pix_colorreduce] is already taken (but then your object is more strict in what it can actually do)
ah, and would it make sense to encode the quality of the guess in the alpha-channel? (or did you plan to output a quality mask as a 2nd image?)
fgmasr IOhannes
--
Ricardo Fabbri
Linux registered user #175401 www.lems.brown.edu/~rfabbri labmacambira.sf.net
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2011-11-11 16:23, Ricardo Fabbri wrote:
sorry for the typos... writing from mobile..
On Friday, November 11, 2011, Ricardo Fabbri rfabbri@gmail.com wrote:
Hello, IOhannes,
glad to hear it can be useful. It could also be called pix_colorclassify
or pix_colorclass, since it outputs 6 color classes 'red', green', ...,
i would go for [pix_colorclassify]. if you agree and are willing to change the name in your branch, i'll import it as is.
'white' (and a seventh 'not sure' class label). However, to me the output is in symbolic form. The 'red' symbol is encoded as [1 0 0] and so on.
makes sense. some more remarks (without having looked at the code in detail, so i might be asking for things already there) - - it would be great if the user could modify those labels. e.g. [1 0 0] sounds like a great label for "red", but some weird user might chose to prefer [0.5 1 0.8].
it would also be nice to be able to parameterize the detection code, but i guess it is too fine tuned to the six given classes (so it would both complicate things from the dev's and the user's povs)
fgmasdr IOhannes
i would go for [pix_colorclassify]. if you agree and are willing to change the name in your branch, i'll import it as is.
I agree and have just renamed it and it compiles cleanly here. You can fetch the changes in the same 'color' branch.
- it would be great if the user could modify those labels. e.g. [1 0 0]
sounds like a great label for "red", but some weird user might chose to prefer [0.5 1 0.8].
That will come with a later contribution. I wrote a [pix_equal] external which can change any pixels "equal" to some value within a given cosed r & g & b range. I use it to separate the colors into different images, change the color label, etc.
it would also be nice to be able to parameterize the detection code, but i guess it is too fine tuned to the six given classes (so it would both complicate things from the dev's and the user's povs)
I guess we should go with the simple algorithm at first, then improve it if needed. I noticed that even with very bad lighting conditions and offset colorbalance, at least 2 or 3 colors work robustly, so this should be useful without tweaking. The parameters have already been tuned on a large database of exemplars. But, I agree, decreasing the sensitivity to e.g. 'red' would be nice. I say we wait for further demands and beta-testing prior to making the external more complex.
Thanks!
Ricardo Fabbri labmacambira.sf.net
On Nov 14, 2011, at 12:30 PM, Ricardo Fabbri wrote:
i would go for [pix_colorclassify]. if you agree and are willing to change the name in your branch, i'll import it as is.
I agree and have just renamed it and it compiles cleanly here. You can fetch the changes in the same 'color' branch.
- it would be great if the user could modify those labels. e.g. [1 0 0]
sounds like a great label for "red", but some weird user might chose to prefer [0.5 1 0.8].
That will come with a later contribution. I wrote a [pix_equal] external which can change any pixels "equal" to some value within a given cosed r & g & b range. I use it to separate the colors into different images, change the color label, etc.
That sounds very useful! I believe that operation is called a color mask. Isn't there something like pix_mask?
.hc
it would also be nice to be able to parameterize the detection code, but i guess it is too fine tuned to the six given classes (so it would both complicate things from the dev's and the user's povs)
I guess we should go with the simple algorithm at first, then improve it if needed. I noticed that even with very bad lighting conditions and offset colorbalance, at least 2 or 3 colors work robustly, so this should be useful without tweaking. The parameters have already been tuned on a large database of exemplars. But, I agree, decreasing the sensitivity to e.g. 'red' would be nice. I say we wait for further demands and beta-testing prior to making the external more complex.
Thanks!
Ricardo Fabbri labmacambira.sf.net
GEM-dev mailing list GEM-dev@iem.at http://lists.puredata.info/listinfo/gem-dev
----------------------------------------------------------------------------
Terrorism is not an enemy. It cannot be defeated. It's a tactic. It's about as sensible to say we declare war on night attacks and expect we're going to win that war. We're not going to win the war on terrorism. - retired U.S. Army general, William Odom
Le 11-11-14 14:15, Hans-Christoph Steiner a écrit :
On Nov 14, 2011, at 12:30 PM, Ricardo Fabbri wrote:
i would go for [pix_colorclassify]. if you agree and are willing to change the name in your branch, i'll import it as is.
I agree and have just renamed it and it compiles cleanly here. You can fetch the changes in the same 'color' branch.
- it would be great if the user could modify those labels. e.g. [1 0 0]
sounds like a great label for "red", but some weird user might chose to prefer [0.5 1 0.8].
That will come with a later contribution. I wrote a [pix_equal] external which can change any pixels "equal" to some value within a given cosed r & g & b range. I use it to separate the colors into different images, change the color label, etc.
That sounds very useful! I believe that operation is called a color mask. Isn't there something like pix_mask?
Yes, it is [pix_bitmask] I use it in this patch. It works well in conjuction with [pix_gain]. I found a reference to it by looking at the opencv objects in Yves' lib, some of which only exist for linux and not OSX. I'm looking forward to new options though, keep up the good work.
Alexandre
.hc
it would also be nice to be able to parameterize the detection code, but i guess it is too fine tuned to the six given classes (so it would both complicate things from the dev's and the user's povs)
I guess we should go with the simple algorithm at first, then improve it if needed. I noticed that even with very bad lighting conditions and offset colorbalance, at least 2 or 3 colors work robustly, so this should be useful without tweaking. The parameters have already been tuned on a large database of exemplars. But, I agree, decreasing the sensitivity to e.g. 'red' would be nice. I say we wait for further demands and beta-testing prior to making the external more complex.
Thanks!
Ricardo Fabbri labmacambira.sf.net
GEM-dev mailing list GEM-dev@iem.at http://lists.puredata.info/listinfo/gem-dev
Terrorism is not an enemy. It cannot be defeated. It's a tactic. It's about as sensible to say we declare war on night attacks and expect we're going to win that war. We're not going to win the war on terrorism. - retired U.S. Army general, William Odom
GEM-dev mailing list GEM-dev@iem.at http://lists.puredata.info/listinfo/gem-dev
On 11/14/2011 06:30 PM, Ricardo Fabbri wrote:
i would go for [pix_colorclassify]. if you agree and are willing to change the name in your branch, i'll import it as is.
I agree and have just renamed it and it compiles cleanly here. You can fetch the changes in the same 'color' branch.
thanks. accepted. and this time i even pushed the changes to sourceforge (which i forgot when importing the [pix_multiblob] changes)...
gmsadr IOhannes
On 11/14/2011 06:30 PM, Ricardo Fabbri wrote:
That will come with a later contribution. I wrote a [pix_equal] external which can change any pixels "equal" to some value within a given cosed r & g & b range. I use it to separate the colors into different images, change the color label, etc.
actually, i think what we really would need was a (high-performant, jit-compiled) [pix_expr] object, that would allow to create such objects in Pd space rather than in C++ space.
fgmasdr IOhannes
-
Alexandre Castonguay -
Hans-Christoph Steiner -
IOhannes m zmoelnig -
IOhannes zmölnig
-
Ricardo Fabbri