Hi, a discussion on facebook led me to revise the help file of [readsf~] and I'm making some edits and changes. It seems it wasn't all to clear how it works, making it seem it would load the file into an internal memory and not just read directly from the disk, and in fact I'm not really sure how it works.
Also, the help has been saying forever how one should open a file "a bit" in advance, which is vague and it also doesn't make it clear why and how long soon... it also says it starts reading from the file right away, but doesn't play it until you say so... this is what makes it a bit confusing to people I guess, cause it seems to load into memory. Now, I believe there is some operation that is done in advance and then it adds some latency maybe, so if you do it in advance you get to manage this a bit better, is that it?
Can we have and add a bit more information about it?
Also, is it the case that this used to be a somewhat significant issue back in the day, which means this has become not significant all for a while?
Last, but not least, I can't make much sense of the 2nd argument. I had a look at the code to see what is the default value (which is also the minimum allowed value), something I always put on the help files, and I wonder if this is some kind of memory buffer that we load the file into... Moreover, why would someone need a bigger buffer than the default value?
I need this information well sorted to make this help file as nice as the others.
Cheers!
Hi,
making it seem it would load the file into an internal memory and not just read directly from the disk
The help file says "soundfile playback from disk"...
Here's how the object works:
A worker thread is reading data from the given file and fills a buffer. If the buffer is full, it waits until there is space. The thread starts to do its job right after the [open( message.
Once we send the [start( message, the perform method simply tries to read a block of samples from the buffer and copy it to the outlets. If there is not enough data in the buffer, the method blocks - which is something we definitely want to avoid! This is exactly the reason why we need to wait a little bit between the [open( message and the [start( message; otherwise the perform routine might have to wait for the buffer, causing a dropout.
The second argument for [readsf~] is the size of the buffer. The default value seems to be 262144 bytes (per channel). In single-precision Pd that corresponds to 65536 samples, which should be more than enough. I think this value comes from the times where everybody had slow HDDs with unpredictable seek times; for modern SSDs it can be much smaller, but we probably don't care about a few kilobytes.
(BTW, I have no idea why the help patch uses a buffer size of 1 MB...)
[writesf~] behaves just the otherway round. Since the perform routine is the producer, we don't need to wait after the [open( message. But after we send [stop(, we should to wait for the worker thread to drain the buffer and write the data to disk before we send another [open( message.
Christof
On 04.03.2024 05:24, Alexandre Torres Porres wrote:
Hi, a discussion on facebook led me to revise the help file of [readsf~] and I'm making some edits and changes. It seems it wasn't all to clear how it works, making it seem it would load the file into an internal memory and not just read directly from the disk, and in fact I'm not really sure how it works.
Also, the help has been saying forever how one should open a file "a bit" in advance, which is vague and it also doesn't make it clear why and how long soon... it also says it starts reading from the file right away, but doesn't play it until you say so... this is what makes it a bit confusing to people I guess, cause it seems to load into memory. Now, I believe there is some operation that is done in advance and then it adds some latency maybe, so if you do it in advance you get to manage this a bit better, is that it?
Can we have and add a bit more information about it?
Also, is it the case that this used to be a somewhat significant issue back in the day, which means this has become not significant all for a while?
Last, but not least, I can't make much sense of the 2nd argument. I had a look at the code to see what is the default value (which is also the minimum allowed value), something I always put on the help files, and I wonder if this is some kind of memory buffer that we load the file into... Moreover, why would someone need a bigger buffer than the default value?
I need this information well sorted to make this help file as nice as the others.
Cheers!
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Actually, I forgot something important:
Of course, the worker thread must also *open* the file! If the file is not yet cached by the OS, this can indeed take a few milliseconds.If you don't add some delay between "open" and "start", you might notice that you get a dropout the very first time, but not on subsequent times.
In fact, if you don't wait between "open" and "start", the perform method almost certainly blocks. However, often we don't notice because it may be "absorbed" by Pd's own ringbuffer (= "Delay" in the audio settings).
Anyway, I agree that the help needs some more clarification! (Just make sure you really understand how the object works before changing the help patch :)
Christof
On 04.03.2024 06:23, Christof Ressi wrote:
Hi,
making it seem it would load the file into an internal memory and not just read directly from the disk
The help file says "soundfile playback from disk"...
Here's how the object works:
A worker thread is reading data from the given file and fills a buffer. If the buffer is full, it waits until there is space. The thread starts to do its job right after the [open( message.
Once we send the [start( message, the perform method simply tries to read a block of samples from the buffer and copy it to the outlets. If there is not enough data in the buffer, the method blocks - which is something we definitely want to avoid! This is exactly the reason why we need to wait a little bit between the [open( message and the [start( message; otherwise the perform routine might have to wait for the buffer, causing a dropout.
The second argument for [readsf~] is the size of the buffer. The default value seems to be 262144 bytes (per channel). In single-precision Pd that corresponds to 65536 samples, which should be more than enough. I think this value comes from the times where everybody had slow HDDs with unpredictable seek times; for modern SSDs it can be much smaller, but we probably don't care about a few kilobytes.
(BTW, I have no idea why the help patch uses a buffer size of 1 MB...)
[writesf~] behaves just the otherway round. Since the perform routine is the producer, we don't need to wait after the [open( message. But after we send [stop(, we should to wait for the worker thread to drain the buffer and write the data to disk before we send another [open( message.
Christof
On 04.03.2024 05:24, Alexandre Torres Porres wrote:
Hi, a discussion on facebook led me to revise the help file of [readsf~] and I'm making some edits and changes. It seems it wasn't all to clear how it works, making it seem it would load the file into an internal memory and not just read directly from the disk, and in fact I'm not really sure how it works.
Also, the help has been saying forever how one should open a file "a bit" in advance, which is vague and it also doesn't make it clear why and how long soon... it also says it starts reading from the file right away, but doesn't play it until you say so... this is what makes it a bit confusing to people I guess, cause it seems to load into memory. Now, I believe there is some operation that is done in advance and then it adds some latency maybe, so if you do it in advance you get to manage this a bit better, is that it?
Can we have and add a bit more information about it?
Also, is it the case that this used to be a somewhat significant issue back in the day, which means this has become not significant all for a while?
Last, but not least, I can't make much sense of the 2nd argument. I had a look at the code to see what is the default value (which is also the minimum allowed value), something I always put on the help files, and I wonder if this is some kind of memory buffer that we load the file into... Moreover, why would someone need a bigger buffer than the default value?
I need this information well sorted to make this help file as nice as the others.
Cheers!
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
Em seg., 4 de mar. de 2024 às 02:25, Christof Ressi info@christofressi.com escreveu:
A worker thread is reading data from the given file and fills a buffer.
and this buffer size is set via the 2nd argument, right?
If the buffer is full, it waits until there is space. The thread starts to do its job right after the [open( message.
And it'll have space when it starts playing, which frees the data from the buffer, huh?
Once we send the [start( message, the perform method simply tries to read a block of samples from the buffer and copy it to the outlets. If there is not enough data in the buffer, the method blocks - which is something we definitely want to avoid! This is exactly the reason why we need to wait a little bit between the [open( message and the [start( message; otherwise the perform routine might have to wait for the buffer, causing a dropout.
But it's not like we need for the whole buffer to get filled before playing, right? So we can read and free from the buffer while it's being filled.
The second argument for [readsf~] is the size of the buffer. The default value seems to be 262144 bytes (per channel). In single-precision Pd that corresponds to 65536 samples, which should be more than enough. I think this value comes from the times where everybody had slow HDDs with unpredictable seek times; for modern SSDs it can be much smaller, but we probably don't care about a few kilobytes.
I see.
(BTW, I have no idea why the help patch uses a buffer size of 1 MB...)
seems weird in fact to require anything bigger than this, really, and this should be clear in the help file! When should one care and feel like 65536 isn't enough?
[writesf~] behaves just the otherway round. Since the perform routine is the producer, we don't need to wait after the [open( message. But after we send [stop(, we should to wait for the worker thread to drain the buffer and write the data to disk before we send another [open( message.
maybe worth mentioning.
Actually, I forgot something important:
Of course, the worker thread must also *open* the file! If the file is not yet cached by the OS, this can indeed take a few milliseconds. If you don't add some delay between "open" and "start", you might notice that you get a dropout the very first time, but not on subsequent times.
In fact, if you don't wait between "open" and "start", the perform method almost certainly blocks. However, often we don't notice because it may be "absorbed" by Pd's own ringbuffer (= "Delay" in the audio settings).
Thing is, I don't think I've ever noticed a dropout and have been ignoring this warning and playing files right away forever, since I started using Pd in the 2000s... I got this rather old 2013 macbook air, and using it with a delay of only 5ms doesn't give me any noticeable dropout!
Can others test this?
This is why I was assuming that it was a concern for pretty old computers. I thought maybe it was a cpu issue, but I see it could also be hard drive related now. This ver old macbook air has got SSDs anyway...
Anyway, I agree that the help needs some more clarification! (Just make sure you really understand how the object works before changing the help patch :)
that's what I am doing :)
Em seg., 4 de mar. de 2024 às 09:08, Dan Wilcox danomatika@gmail.com escreveu:
*This* is a good point and worth noting, not necessarily as a "always delay before playing" sort of thing but more a short description of how it works, then a note like "if you experience occasional dropouts on first accessing a file, consider adding a small delay after opening but *before* playing."
nice suggestion :) thanks
Em qua., 6 de mar. de 2024 às 21:27, Alexandre Torres Porres < porres@gmail.com> escreveu:
I got this rather old 2013 macbook air, and using it with a delay of only 5ms doesn't give me any noticeable dropout!
btw, 4ms chokes and causes dropouts when testing audio with the audio-mid test patch, just playing a sine wave...
On 07.03.2024 01:27, Alexandre Torres Porres wrote:
Em seg., 4 de mar. de 2024 às 02:25, Christof Ressi info@christofressi.com escreveu:
A worker thread is reading data from the given file and fills a buffer.and this buffer size is set via the 2nd argument, right?
Yes!
If the buffer is full, it waits until there is space. The thread starts to do its job right after the [open( message.And it'll have space when it starts playing, which frees the data from the buffer, huh?
Yes, although "frees the data" is the wrong term. The perform routine reads a range of samples and advances the read pointer, after which these samples can be *overwritten* by the worker thread. It's just a simple ringbuffer.
Once we send the [start( message, the perform method simply tries to read a block of samples from the buffer and copy it to the outlets. If there is not enough data in the buffer, the method blocks - which is something we definitely want to avoid! This is exactly the reason why we need to wait a little bit between the [open( message and the [start( message; otherwise the perform routine might have to wait for the buffer, causing a dropout.But it's not like we need for the whole buffer to get filled before playing, right? So we can read and free from the buffer while it's being filled.
Yes! Although in practice the buffer will be filled pretty quickly once the soundfile has been opened. One could say that the perform routine "drives" the worker thread.
The second argument for [readsf~] is the size of the buffer. The default value seems to be 262144 bytes (per channel). In single-precision Pd that corresponds to 65536 samples, which should be more than enough. I think this value comes from the times where everybody had slow HDDs with unpredictable seek times; for modern SSDs it can be much smaller, but we probably don't care about a few kilobytes.I see.
(BTW, I have no idea why the help patch uses a buffer size of 1 MB...)seems weird in fact to require anything bigger than this, really, and this should be clear in the help file! When should one care and feel like 65536 isn't enough?
Pretty much never.
[writesf~] behaves just the otherway round. Since the perform routine is the producer, we don't need to wait after the [open( message. But after we send [stop(, we should to wait for the worker thread to drain the buffer and write the data to disk before we send another [open( message.maybe worth mentioning.
Actually, I forgot something important: Of course, the worker thread must also *open* the file! If the file is not yet cached by the OS, this can indeed take a few milliseconds.If you don't add some delay between "open" and "start", you might notice that you get a dropout the very first time, but not on subsequent times. In fact, if you don't wait between "open" and "start", the perform method almost certainly blocks. However, often we don't notice because it may be "absorbed" by Pd's own ringbuffer (= "Delay" in the audio settings).Thing is, I don't think I've ever noticed a dropout and have been ignoring this warning and playing files right away forever, since I started using Pd in the 2000s... I got this rather old 2013 macbook air, and using it with a delay of only 5ms doesn't give me any noticeable dropout!
I did and still do notice it! Opening a file (that is not cached by the OS) can take a few milliseconds. If your delay setting is low and CPU load is high, you can easily get a dropout.
Can others test this?
This is why I was assuming that it was a concern for pretty old computers. I thought maybe it was a cpu issue, but I see it could also be hard drive related now. This ver old macbook air has got SSDs anyway...
Anyway, I agree that the help needs some more clarification! (Just make sure you really understand how the object works before changing the help patch :)that's what I am doing :)
+1
Em qua., 6 de mar. de 2024 às 21:43, Christof Ressi info@christofressi.com escreveu:
The perform routine reads a range of samples and advances the read pointer, after which these samples can be *overwritten* by the worker thread. It's just a simple ringbuffer.
Got it!
I did and still do notice it! Opening a file (that is not cached by the OS) can take a few milliseconds. If your delay setting is low and CPU load is high, you can easily get a dropout.
Ok :)
Anyway, here's my current revision, uploading it right now...
[image: Screen Shot 2024-03-06 at 21.46.22.png]
Anyway, here's my current revision, uploading it right now...
Screen Shot 2024-03-06 at 21.46.22.png
Suggestion: "The [readsf~] object streams a soundfile from the hard disk."
Suggestion: "You should wait a few milliseconds between "open" and "start" to ensure that the buffer is filled in time, otherwise you may get a dropout."
Em qua., 6 de mar. de 2024 às 22:10, Christof Ressi info@christofressi.com escreveu:
Suggestion: "You should wait a few milliseconds between "open" and "start" to ensure that the buffer is filled in time, otherwise you may get a dropout."
How about "*You should wait a few milliseconds between "open" and "start" on first accessing a file to ensure that the buffer is filled in time, otherwise you may get a dropout.*"?
Pd-list@lists.iem.at mailing list UNSUBSCRIBE and account-management -> https://lists.puredata.info/listinfo/pd-list
On 07.03.2024 02:38, Alexandre Torres Porres wrote:
on first accessing a file
I think this would cause more confusion than it would help. (What does "first accessing" actually mean?)
I wrote about the case where the file is not yet cached by the OS, but IMO that is too specific for a help patch. Also, it isn't the *only* case that can cause a dropout. In general, file I/O is non-deterministic and you are at the mercy of your OS.
I think it's enough to tell users that the buffer needs to be filled in time.
Christof
thing is I can never hear dropouts, don't think I've ever had problems, then it seemed that the issue might be first accessing an uncached file, which kinda made sense why I never faced this.... I also have an abstraction that is a wrap around readsf~ that doesn't care about this and I use it many times to randomly play samples from sample banks, never found anything funny (ok, my performances are usually loud and noisy anyway, haha)
And for testing now, I just recorded a new file (ok it was in Pd), then renamed it and moved it elsewhere, no dropouts either, did my system cash this somehow and kept track of it?
Anyway, I really like how Dan put it... which is you "may" get dropouts... "if you do" then you should do this... because putting this as something thas *has* to be done everytime doesn't seem right... :)
Em qua., 6 de mar. de 2024 às 23:33, Christof Ressi info@christofressi.com escreveu:
On 07.03.2024 02:38, Alexandre Torres Porres wrote:
on first accessing a file
I think this would cause more confusion than it would help. (What does "first accessing" actually mean?)
I wrote about the case where the file is not yet cached by the OS, but IMO that is too specific for a help patch. Also, it isn't the *only* case that can cause a dropout. In general, file I/O is non-deterministic and you are at the mercy of your OS.
I think it's enough to tell users that the buffer needs to be filled in time.
Christof
On 07.03.2024 04:05, Alexandre Torres Porres wrote:
thing is I can never hear dropouts, don't think I've ever had problems, then it seemed that the issue might be first accessing an uncached file, which kinda made sense why I never faced this.... I also have an abstraction that is a wrap around readsf~ that doesn't care about this and I use it many times to randomly play samples from sample banks, never found anything funny (ok, my performances are usually loud and noisy anyway, haha)
I definitely did get dropouts. Maybe you just never noticed it :-D Or macOS has just faster file I/O than Windows (which I believe is indeed the case).
And for testing now, I just recorded a new file (ok it was in Pd), then renamed it and moved it elsewhere, no dropouts either, did my system cash this somehow and kept track of it?
Yes, very likely.
Anyway, I really like how Dan put it... which is you "may" get dropouts... "if you do" then you should do this... because putting this as something thas *has* to be done everytime doesn't seem right... :)
I don't agree. The problem with "if you do" is that it isn't portable. If you don't care about writing portable patches, that's probably fine, but I think we should really teach users the correct and portable way. The very idea of [readsf~] is that you don't open the file on the audio thread, but when you do [open <f>, start( you indirectly block the audio thread. Just don't do it!
I've already mentioned this, but if you do [open <f>, start( it can become quite awkward to change later. Better do the right thing from the beginning.
(BTW, the equivalent SuperCollider object would be DiskIn UGen + Buffer.cueSoundFile; there you don't even have the chance to skip the buffering step.)
Christof
Em qua., 6 de mar. de 2024 às 23:33, Christof Ressi info@christofressi.com escreveu:
On 07.03.2024 02:38, Alexandre Torres Porres wrote: > on first accessing a file I think this would cause more confusion than it would help. (What does "first accessing" actually mean?) I wrote about the case where the file is not yet cached by the OS, but IMO that is too specific for a help patch. Also, it isn't the *only* case that can cause a dropout. In general, file I/O is non-deterministic and you are at the mercy of your OS. I think it's enough to tell users that the buffer needs to be filled in time. Christof
ok, I'm working a bit more on it and will take your suggestion, which is " "You should wait a few milliseconds between "open" and "start" to ensure that the buffer is filled in time, otherwise you may get a dropout."" or something like that...
Em qui., 7 de mar. de 2024 às 00:27, Christof Ressi info@christofressi.com escreveu:
On 07.03.2024 04:05, Alexandre Torres Porres wrote:
thing is I can never hear dropouts, don't think I've ever had problems, then it seemed that the issue might be first accessing an uncached file, which kinda made sense why I never faced this.... I also have an abstraction that is a wrap around readsf~ that doesn't care about this and I use it many times to randomly play samples from sample banks, never found anything funny (ok, my performances are usually loud and noisy anyway, haha)
I definitely did get dropouts. Maybe you just never noticed it :-D Or macOS has just faster file I/O than Windows (which I believe is indeed the case).
And for testing now, I just recorded a new file (ok it was in Pd), then renamed it and moved it elsewhere, no dropouts either, did my system cash this somehow and kept track of it?
Yes, very likely.
Anyway, I really like how Dan put it... which is you "may" get dropouts... "if you do" then you should do this... because putting this as something thas *has* to be done everytime doesn't seem right... :)
I don't agree. The problem with "if you do" is that it isn't portable. If you don't care about writing portable patches, that's probably fine, but I think we should really teach users the correct and portable way. The very idea of [readsf~] is that you don't open the file on the audio thread, but when you do [open <f>, start( you indirectly block the audio thread. Just don't do it!
I've already mentioned this, but if you do [open <f>, start( it can become quite awkward to change later. Better do the right thing from the beginning.
(BTW, the equivalent SuperCollider object would be DiskIn UGen + Buffer.cueSoundFile; there you don't even have the chance to skip the buffering step.) Christof
Em qua., 6 de mar. de 2024 às 23:33, Christof Ressi < info@christofressi.com> escreveu:
On 07.03.2024 02:38, Alexandre Torres Porres wrote:
on first accessing a file
I think this would cause more confusion than it would help. (What does "first accessing" actually mean?)
I wrote about the case where the file is not yet cached by the OS, but IMO that is too specific for a help patch. Also, it isn't the *only* case that can cause a dropout. In general, file I/O is non-deterministic and you are at the mercy of your OS.
I think it's enough to tell users that the buffer needs to be filled in time.
Christof
Em qui., 7 de mar. de 2024 às 00:27, Christof Ressi info@christofressi.com escreveu:
(BTW, the equivalent SuperCollider object would be DiskIn UGen + Buffer.cueSoundFile; there you don't even have the chance to skip the buffering step.)
This is a much better design. I don't mind the latency, but I'd really like to tell the object to just play and it can then work out when it is ready and start doing it :)
In fact, I would like to suggest a new "play" message, that would do exactly this: "open" + "start', making sure there are no dropouts...
Seems like a job to you by the way ;)
On 07.03.2024 04:43, Alexandre Torres Porres wrote:
Em qui., 7 de mar. de 2024 às 00:27, Christof Ressi info@christofressi.com escreveu:
(BTW, the equivalent SuperCollider object would be DiskIn UGen + Buffer.cueSoundFile; there you don't even have the chance to skip the buffering step.)This is a much better design. I don't mind the latency, but I'd really like to tell the object to just play and it can then work out when it is ready and start doing it :)
Keep in mind that this is not deterministic, i.e. playback will just start whenever the data is ready. This is probably fine for many uses cases, but if you want sample accurate playback, you'd need to wait until Buffer.cueSoundFile has completed before creating the DiskIn, similar to [readsf~].
In fact, I would like to suggest a new "play" message, that would do exactly this: "open" + "start', making sure there are no dropouts...
Actually, I had already thought about that! There could be an option that makes [readsf~] non-blocking, i.e. the perform routine does not wait on a condition variable if the buffer is empty, instead it would just output silence. This would be essentially the same behavior as DiskIn. I thought of doing this with a flag to the "open" message: [open -n <foo>, start(
But I agree that [play <foo>( looks much nicer and less obscure.
I just opened a feature request: https://github.com/pure-data/pure-data/issues/2205 :)
Seems like a job to you by the way ;)
-
Alexandre Torres Porres -
Christof Ressi