I've been studying the work of Neil Harbisson for awhile and I'm looking to try and replicate in part the functionality of his eyeborg using Pure Data. I have already built a simple patch to convert pixels to sound but now I want to expand on it. Here's my initial research: http://www.hellocatfood.com/sonification-studiespixel-waves/
Colours don't directly relate to sound and so Harbisson and others must use a scale to assign colours to sounds. Some initial research brought these up:
http://blogs.plos.org/neuroanthropology/files/2014/03/Harbisson-The-Sound-of... http://www.bbc.co.uk/news/magazine-16681630 http://www.lunarplanner.com/Harmonics/planetary-harmonics.html http://www.flutopedia.com/sound_color.htm
Can anyone think of a way to translate this into pd? In the end I would like to be able to display a block of colour on screen and have that generate a specific note.
Any help is appreciated.
Antonio
Hi Antonio,
On 05/04/15 15:51, Antonio Roberts wrote:
I've been studying the work of Neil Harbisson for awhile and I'm looking to try and replicate in part the functionality of his eyeborg using Pure Data. I have already built a simple patch to convert pixels to sound but now I want to expand on it. Here's my initial research: http://www.hellocatfood.com/sonification-studiespixel-waves/
"I predict the development of an image to audio sequencer in the near future."
I made a painting program in gridflow years ago, had some very basic sonification - gridflow also had a "how to play a car" example converting a photo by scanlines into audio with FFT.
https://archive.org/details/ClaudiusMaximus_-_Emulsion https://archive.org/details/ClaudiusMaximus_-_DohPaintII_Session_3 https://archive.org/details/ClaudiusMaximus_-_DohPaintII_Session_2
Colours don't directly relate to sound and so Harbisson and others must use a scale to assign colours to sounds. Some initial research brought these up:
http://blogs.plos.org/neuroanthropology/files/2014/03/Harbisson-The-Sound-of... http://www.bbc.co.uk/news/magazine-16681630 http://www.lunarplanner.com/Harmonics/planetary-harmonics.html http://www.flutopedia.com/sound_color.htm
The last one, in the section "Converting RGB Colors to Color Frequencies", makes a good point - monochromatic wavelength to RGB works fine, but RGB to wavelength is in general impossible, as colours are spectral power distributions over continuous wavelengths. Once it's flattened to RGB, too much information is lost to be able to recreate the original spectrum (aka metamerism, where 2 different spectrum give same colour sensation).
Recently I copy/pasted some colour-related stuff from Wikipedia into a booklet (second link is layed out for printing, first one is better for screen reading, both have the same content):
http://mathr.co.uk/misc/2015-04-04_colour.pdf http://mathr.co.uk/misc/2015-04-04_colour_booklet.pdf
The approaches you linked seem to convert wavelengths to colour in a straightforward way, but for converting from colour to sound I think a different approach would be better (and indeed needed) - something more akin to the Munsell perceptual colour system:
https://en.wikipedia.org/wiki/Munsell_color_system
a grossly simplified version would be HSV, which I used in the attached
- a bank of vcf~ fed by noise~, mapping hue to scale note, value to
loudness and pitch, and saturation to filter Q - so that white would give high noise, grey gives a lower pitch noise, black gives silence, fully-saturated dark green gives a low note, bright green gives a high pitch at the same scale note. the filters glitch when the hue jumps from 0 red (orange side) to 1 red (purple side), could be fixed with some cleverness probably.
Can anyone think of a way to translate this into pd? In the end I would like to be able to display a block of colour on screen and have that generate a specific note.
Any help is appreciated.
Antonio
Claude
Hi Claude, Interesting stuff! I triend to run your patch, but I get "./recog~: can't load library" (and pd seems to go into a "watchdog: signaling pd..." loop as soon as I create the Gem window, don't know if that's related). I can't find 'recog~' anywhere on my system. hint welcome. thanks, Tim
2015-04-07 17:58 GMT+02:00 Claude Heiland-Allen claude@mathr.co.uk:
Hi Antonio,
On 05/04/15 15:51, Antonio Roberts wrote:
I've been studying the work of Neil Harbisson for awhile and I'm looking to try and replicate in part the functionality of his eyeborg using Pure Data. I have already built a simple patch to convert pixels to sound but now I want to expand on it. Here's my initial research: http://www.hellocatfood.com/sonification-studiespixel-waves/
"I predict the development of an image to audio sequencer in the near future."
I made a painting program in gridflow years ago, had some very basic sonification - gridflow also had a "how to play a car" example converting a photo by scanlines into audio with FFT.
https://archive.org/details/ClaudiusMaximus_-_Emulsion https://archive.org/details/ClaudiusMaximus_-_DohPaintII_Session_3 https://archive.org/details/ClaudiusMaximus_-_DohPaintII_Session_2
Colours don't directly relate to sound and so Harbisson and others
must use a scale to assign colours to sounds. Some initial research brought these up:
http://blogs.plos.org/neuroanthropology/files/2014/ 03/Harbisson-The-Sound-of-Colors-TED.jpg http://www.bbc.co.uk/news/magazine-16681630 http://www.lunarplanner.com/Harmonics/planetary-harmonics.html http://www.flutopedia.com/sound_color.htm
The last one, in the section "Converting RGB Colors to Color Frequencies", makes a good point - monochromatic wavelength to RGB works fine, but RGB to wavelength is in general impossible, as colours are spectral power distributions over continuous wavelengths. Once it's flattened to RGB, too much information is lost to be able to recreate the original spectrum (aka metamerism, where 2 different spectrum give same colour sensation).
Recently I copy/pasted some colour-related stuff from Wikipedia into a booklet (second link is layed out for printing, first one is better for screen reading, both have the same content):
http://mathr.co.uk/misc/2015-04-04_colour.pdf http://mathr.co.uk/misc/2015-04-04_colour_booklet.pdf
The approaches you linked seem to convert wavelengths to colour in a straightforward way, but for converting from colour to sound I think a different approach would be better (and indeed needed) - something more akin to the Munsell perceptual colour system:
https://en.wikipedia.org/wiki/Munsell_color_system
a grossly simplified version would be HSV, which I used in the attached - a bank of vcf~ fed by noise~, mapping hue to scale note, value to loudness and pitch, and saturation to filter Q - so that white would give high noise, grey gives a lower pitch noise, black gives silence, fully-saturated dark green gives a low note, bright green gives a high pitch at the same scale note. the filters glitch when the hue jumps from 0 red (orange side) to 1 red (purple side), could be fixed with some cleverness probably.
Can anyone think of a way to translate this into pd? In the end I would like to be able to display a block of colour on screen and have that generate a specific note.
Any help is appreciated.
Antonio
Claude
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
On 07/04/15 17:35, tim vets wrote:
I get "./recog~: can't load library" (and pd seems to go into a "watchdog: signaling pd..." loop as soon as I create the Gem window, don't know if that's related). I can't find 'recog~' anywhere on my system.
I suspect a problem with your Gem rather than my simple patch... does your Gem work with other patches?
2015-04-09 17:58 GMT+02:00 Claude Heiland-Allen claude@mathr.co.uk:
On 07/04/15 17:35, tim vets wrote:
I get "./recog~: can't load library" (and pd seems to go into a "watchdog: signaling pd..." loop as soon as I create the Gem window, don't know if that's related). I can't find 'recog~' anywhere on my system.
I suspect a problem with your Gem rather than my simple patch... does your Gem work with other patches?
Yes, Gem works, pd 0.43.4-extended, maybe that's too old a version?
Tim
Greetings Ant!
Perhaps try looking at [pix_mean_color] object? Attached is a patch that I've cleaned up- it just extracts RGA values from loaded video then unpacks them for 3 [osc~] objects-crude but could be starting point I hope
...best!
Al
On Sun, Apr 5, 2015 at 3:51 PM, Antonio Roberts antonio@hellocatfood.com wrote:
I've been studying the work of Neil Harbisson for awhile and I'm looking to try and replicate in part the functionality of his eyeborg using Pure Data. I have already built a simple patch to convert pixels to sound but now I want to expand on it. Here's my initial research: http://www.hellocatfood.com/sonification-studiespixel-waves/
Colours don't directly relate to sound and so Harbisson and others must use a scale to assign colours to sounds. Some initial research brought these up:
http://blogs.plos.org/neuroanthropology/files/2014/03/Harbisson-The-Sound-of... http://www.bbc.co.uk/news/magazine-16681630 http://www.lunarplanner.com/Harmonics/planetary-harmonics.html http://www.flutopedia.com/sound_color.htm
Can anyone think of a way to translate this into pd? In the end I would like to be able to display a block of colour on screen and have that generate a specific note.
Any help is appreciated.
Antonio
--
antonio@hellocatfood.com http://www.hellocatfood.com ============================
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Hi Alan and Claude,
Thanks for your insights. I now understand that there is no one "true" way to convert colour to sound and that there will always be a "tuning bias", similar to what I already did in my Pixel Player software.
Thanks for the heads up about the [pix_mean_color] object, I think I'll incorporate it into a new version of the software.
Antonio
On 9 April 2015 at 21:20, Alan Brooker alan.brooker2010@gmail.com wrote:
Greetings Ant!
Perhaps try looking at [pix_mean_color] object? Attached is a patch that I've cleaned up- it just extracts RGA values from loaded video then unpacks them for 3 [osc~] objects-crude but could be starting point I hope
...best!
Al
On Sun, Apr 5, 2015 at 3:51 PM, Antonio Roberts antonio@hellocatfood.com wrote:
I've been studying the work of Neil Harbisson for awhile and I'm looking to try and replicate in part the functionality of his eyeborg using Pure Data. I have already built a simple patch to convert pixels to sound but now I want to expand on it. Here's my initial research: http://www.hellocatfood.com/sonification-studiespixel-waves/
Colours don't directly relate to sound and so Harbisson and others must use a scale to assign colours to sounds. Some initial research brought these up:
http://blogs.plos.org/neuroanthropology/files/2014/03/Harbisson-The-Sound-of... http://www.bbc.co.uk/news/magazine-16681630 http://www.lunarplanner.com/Harmonics/planetary-harmonics.html http://www.flutopedia.com/sound_color.htm
Can anyone think of a way to translate this into pd? In the end I would like to be able to display a block of colour on screen and have that generate a specific note.
Any help is appreciated.
Antonio
--
antonio@hellocatfood.com http://www.hellocatfood.com ============================
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
Hi,
I've been looking for a ready-made solution to make Two to many mapping like the [nodes] object from Max,
but I couldn't find equivalent on PD,
have you heard about something like that ? I try to use some perceptron model from the ml.lib but that's way too a bit too big for me.
Thanks !
P.
Hi all,
Thanks for the great info. [pix_mean_color] is awesome! I was looking for something like this for a patch. My patch is very similar to the ones in this thread, but I'm looking to have multiple synths generating sound using different areas of the movie or picture. I'd REALLY love if I could specify a polygon, single pixel, or set of pixels (arbitrary or along a ling segment or curve) within the picture, for [pix_mean_color] to use.
I've been searching through the gem objects for a way to limit which pixels [pix_mean_color] will see. I imagine another possibility is to show limited areas of the same video, each in a different patch, and then use [pix_mean_color] as in patches above. I'm not experience with gem and can't imagine how to do either. Has anyone done this?
Best,
Chris
On Sun, Apr 12, 2015 at 11:54 AM, Antonio Roberts antonio@hellocatfood.com wrote:
Hi Alan and Claude,
Thanks for your insights. I now understand that there is no one "true" way to convert colour to sound and that there will always be a "tuning bias", similar to what I already did in my Pixel Player software.
Thanks for the heads up about the [pix_mean_color] object, I think I'll incorporate it into a new version of the software.
Antonio
On 9 April 2015 at 21:20, Alan Brooker alan.brooker2010@gmail.com wrote:
Greetings Ant!
Perhaps try looking at [pix_mean_color] object? Attached is a patch that I've cleaned up- it just extracts RGA values from loaded video then
unpacks
them for 3 [osc~] objects-crude but could be starting point I hope
...best!
Al
On Sun, Apr 5, 2015 at 3:51 PM, Antonio Roberts <
antonio@hellocatfood.com>
wrote:
I've been studying the work of Neil Harbisson for awhile and I'm looking to try and replicate in part the functionality of his eyeborg using Pure Data. I have already built a simple patch to convert pixels to sound but now I want to expand on it. Here's my initial research: http://www.hellocatfood.com/sonification-studiespixel-waves/
Colours don't directly relate to sound and so Harbisson and others must use a scale to assign colours to sounds. Some initial research brought these up:
http://blogs.plos.org/neuroanthropology/files/2014/03/Harbisson-The-Sound-of...
http://www.bbc.co.uk/news/magazine-16681630 http://www.lunarplanner.com/Harmonics/planetary-harmonics.html http://www.flutopedia.com/sound_color.htm
Can anyone think of a way to translate this into pd? In the end I would like to be able to display a block of colour on screen and have that generate a specific note.
Any help is appreciated.
Antonio
--
antonio@hellocatfood.com http://www.hellocatfood.com ============================
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
--
antonio@hellocatfood.com http://www.hellocatfood.com ============================
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> http://lists.puredata.info/listinfo/pd-list
-
Alan Brooker -
Antonio Roberts -
Christopher Meng -
Claude Heiland-Allen -
Pierre Desprats -
tim vets