Something struck me whilst out walking today; has anyone applied the seam carving technique:
http://en.wikipedia.org/wiki/Seam_carving <www.youtube.com/watch?v=6NcIJXTlugc>
to FFT data to do a timestretch/timecompress that is more sensitive to attack sounds and note lengths?
Best,
Chris.
On Mon, 9 Jun 2008, Chris McCormick wrote:
Something struck me whilst out walking today; has anyone applied the seam carving technique: http://en.wikipedia.org/wiki/Seam_carving <www.youtube.com/watch?v=6NcIJXTlugc> to FFT data to do a timestretch/timecompress that is more sensitive to attack sounds and note lengths?
I doubt that this would work at all. It's already of dubious use on images, as it introduces distorsions that even though they are made to look least like distorsions, are serious distorsions anyway, and can mess up the perception of the content seriously. Compared to the "deleting least important columns" technique that they show at 1:40, the Seam Carving technique sweeps the distorsions under the carpet. After that, don't wonder what the big lump under the carpet is.
When applying the technique to the spectrogram of a song, you should hear it many times more than you'd see it.
Perhaps it can be used as an interesting special effect, supposing that what you want to do is mess up the rhythm seriously and turn chords into arpeggios in truly weird ways. However, it will work at the sine wave level, which is somewhat less attractive than if it worked on notes.
Instead of applying it on FFT, you could apply it on a piano roll. (A real one, unlike GridFlow's patch of that name, which is actually botched FFT synthesis)
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
On Tue, Jul 08, 2008 at 02:53:56AM -0400, Mathieu Bouchard wrote:
On Mon, 9 Jun 2008, Chris McCormick wrote:
Something struck me whilst out walking today; has anyone applied the seam carving technique: http://en.wikipedia.org/wiki/Seam_carving <www.youtube.com/watch?v=6NcIJXTlugc> to FFT data to do a timestretch/timecompress that is more sensitive to attack sounds and note lengths?
Compared to the "deleting least important columns" technique that they show at 1:40, the Seam Carving technique sweeps the distorsions under the carpet. After that, don't wonder what the big lump under the carpet is.
Yes, that is the exact point of seam carving. I'm not sure why you're putting a negative lumps-under-the-carpet slant on the idea.
When applying the technique to the spectrogram of a song, you should hear it many times more than you'd see it.
I take it this is speculation. I am interested to know if anyone has tried it and listened to the results.
Perhaps it can be used as an interesting special effect, supposing that what you want to do is mess up the rhythm seriously and turn chords into arpeggios in truly weird ways. However, it will work at the sine wave level, which is somewhat less attractive than if it worked on notes.
Isn't part of what makes a note perceptible it's higher energy compared to the surrounding space in the spectrogram? If this is the case, seam carving would delete seams around the notes since they have less energy, whilst preserving the sounds of the notes themselves. This might sound like a mess, but I'd like to hear it.
Maybe I sould just do it.
Chris.
On Tue, 8 Jul 2008, Chris McCormick wrote:
Compared to the "deleting least important columns" technique that they show at 1:40, the Seam Carving technique sweeps the distorsions under the carpet. After that, don't wonder what the big lump under the carpet is.
Yes, that is the exact point of seam carving. I'm not sure why you're putting a negative lumps-under-the-carpet slant on the idea.
Well, it's only negative if someone is going to be using this as if it were for _scaling_ the picture. If from the start it's clear that this is going to be an effect that is quite special, it's alright. I wouldn't like to have this to resize images in a web browser, for example.
When applying the technique to the spectrogram of a song, you should hear it many times more than you'd see it.
I take it this is speculation. I am interested to know if anyone has tried it and listened to the results.
No, I haven't listened to it. So I suppose that what I say doesn't count.
Isn't part of what makes a note perceptible it's higher energy compared to the surrounding space in the spectrogram?
Yes, but it all depends on how you define "energy" for the occasion. Throughout the video, the concept of energy is redefined (e.g.: to edit people out), but even their original definition is somewhat special-purpose, just like it usually is in image processing... I mean that there is no standard definition for it in image processing.
Maybe I sould just do it.
You should do it because you will not hear about anyone that has actually tried it, and you want to know what it is really like, not what I think it will do. I still recommend that you do it on an actual score instead of (or in addition to) a spectrogram.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
Chris McCormick schreef:
Isn't part of what makes a note perceptible it's higher energy compared to the surrounding space in the spectrogram? If this is the case, seam carving would delete seams around the notes since they have less energy, whilst preserving the sounds of the notes themselves. This might sound like a mess, but I'd like to hear it.
Maybe I sould just do it.
that's the spirit. i reckon it could sound super-amazing. let me know if you give it a go, and pass on a link to the audio results...
d
Chris McCormick schreef:
This might sound like a mess, but I'd like to hear it.
(i'm reminded of a particular electroacoustic composition tutorial class i attended when studying a couple of years ago, where we were being taught to use SoundHack and the tutor was saying all sorts of things to us like 'you should use the Hanning window here because it sounds best' and 'don't use settings here below <x> because it won't sound good'. i managed to convince him to demonstrate what 'sounding bad' was like, and then we spent the rest of the tutorial playing with settings and making 'bad sounding' audio. it was great.)
d
i'm thinking about how you'd do this,
and the picture in my mind is of a single 2 second long clarinet note with a hihat being hit after about 1 second.
so, if you were going to stretch or compress that, you'd want to preserve the parts with the most 'colour' ...which i would guess would be the attack of the clarinet and then especially the attack of the hihat.
[bonk~] will find those bits for you quite easily. so you can just mark the sections bonk~ finds and set them as 'do not touch'
but the hard part is stretching the remaining sections. i think ableton and melodyne have good approaches for this, and from what i can gather it's a good granular approach with the grain length being constantly adjusted so that the playback doesn't introduce too much AM effect.
i have a feeling they also do a pitch analysis, so that the playback grain length can be adjusted longer for lower overall pitch and shorter for higher pitched sections.
i could be completely wrong of course. wouldn't be the first time. :)
- On 2008-07-08 hard off hard.off@gmail.com wrote :
i'm thinking about how you'd do this,
and the picture in my mind is of a single 2 second long clarinet note with a hihat being hit after about 1 second.
so, if you were going to stretch or compress that, you'd want to preserve the parts with the most 'colour' ...which i would guess would be the attack of the clarinet and then especially the attack of the hihat.
Maybe a bit too naive approach, but what about chopping up the audio in small fragments, and discarts fragments that have about the same spectral footprint as their neighbours ? This will leave the dynamic parts intact and leaves out the 'boring' bits.
Ico
On Tue, 8 Jul 2008, Damian Stewart wrote:
(i'm reminded of a particular electroacoustic composition tutorial class i attended when studying a couple of years ago, where we were being taught to use SoundHack and the tutor was saying all sorts of things to us like 'you should use the Hanning window here because it sounds best' and 'don't use settings here below <x> because it won't sound good'. i managed to convince him to demonstrate what 'sounding bad' was like, and then we spent the rest of the tutorial playing with settings and making 'bad sounding' audio. it was great.)
Yes. As I said, I'm sure you can get pretty cool sounds out of this effect, but if all you're looking for is a way to make a cleaner kind of stretching/shrinking, it's the wrong place to look for it, which is why I was being negative about it. If the point is to make funny-sounding stuff, you probably have something more interesting than a shovelful of allpass filtres here.
On a related note, gridflow/examples/postdigital.pd is an example of messing with a setting called "quality", _lowering_ it until it looks so low-fidelity that it becomes _interesting_. Then it shows the frame difference with the original image, which is even more fun. I think of postdigitalism as about not taking quality settings literally, which is why I came up with this patch under that name, or came up with that name for that patch.
Note that my use of the term is tongue-in-cheek and I haven't read Kim Cascone nor Roy Ascott, and chances are that I'm not going to do it soon (I have a large pile of unread books in the queue and it doesn't get any smaller). But I enjoy talking about those æsthetical issues a lot, just that I tend to use my existing background instead, which may give wholly different results... e.g.: what if Claude Shannon had written a book named «Différence et Répétition»? that's the kind of question I sometimes ask myself.
_ _ __ ___ _____ ________ _____________ _____________________ ... | Mathieu Bouchard - tél:+1.514.383.3801, Montréal, Québec
-
Chris McCormick -
Damian Stewart -
hard off -
Ico Doornekamp -
Mathieu Bouchard